July 2020 - Dev - mailweb.openlookeng.io

[all]PPT templates are available
by Fred Li 17 Jul '20

17 Jul '20

Hi community, Thanks to lvzixin who helped us to design the ppt templates[1]. [1] https://gitee.com/openlookeng/community/tree/master/artwork/ppt_template -- Regards Fred Li (李永乐)

1 0

openLooKeng PMC Biweekly Meeting
by Maowei (Morewell) 16 Jul '20

16 Jul '20

openLooKeng PMC Biweekly Meeting Agenda: * PMC Operation Mechanism * Large-granularity Milesetone & RoadMap * 930 openLooKeng version features * Other…… If you have other topics to discuss please leave a message here：https://etherpad.openlookeng.io/p/PMC-weeklymeeting Meeting Link: https://welink-meeting.zoom.us/j/813957346

1 0

Auto cleanup for Hive ORC compaction
by Aman Omer 15 Jul '20

15 Jul '20

Hi all, Since, openlookeng is now supporting compaction/vacuum operation on Hive transactional tables which merges all base and delta files together (according to the compaction type). So that upcoming read queries can be redirected to use newly created, compacted directories which improves performance. This makes old, smaller delta directories eligible for cleanup. Therefore, a background process is required which will be responsible for removing any redundant directory from table path. Check attached document for more details. Kindly let me know if there are any comments/suggestions. I have created a slack channel (#auto-cleanup). Kindly join it for discussion. Thanks, Aman Omer

1 0

Carbondata Auto-Cleanup
by K Sandeep 15 Jul '20

15 Jul '20

Hello all, Openlookeng integration with Carbondata not supporting auto cleanup of vacuum/compaction tables. This becomes the limitation of openlookeng since other processing engine like spark provides the same. Find initial design doc in attachments. Kindly let me know if there are any comments/suggestions. Thanks, Sandeep.k

1 0

Carbondata Auto-Cleanup
by Sandeep K 15 Jul '20

15 Jul '20

Hello all, Openlookeng integration with Carbondata not supporting auto cleanup of vacuum/compaction tables. This becomes the limitation of openlookeng since other processing engine like spark provides the same. Find initial design doc in attachments. Kindly let me know if there are any comments/suggestions. Thanks, Sandeep.k

1 0

IUD support on Carbondata partitioned table
by Aman Omer 15 Jul '20

15 Jul '20

Hello all, Openlookeng integration with Carbondata supports creation and reading data from partitioned table but doesn’t provide support for insert, update and delete. This becomes the limitation of openlookeng since other processing engine like spark provides the same. Insert, Update and Delete support on carbondata partitioned table is the high level goal of this proposal. Find initial design doc in attachments. Kindly let me know if there are any comments/suggestions. Thanks, Aman Omer

2 1

Welcome Xulifeng and Xutianli to the openLooKeng PMC
by Ken Zhang 15 Jul '20

15 Jul '20

Hi All, Our community is expanding! To ensure our roadmap and vision matches the big data trend. I am honored to announce Xulifeng and Xutianli has accepted our invitation to join openLooKeng PMC. Both of them have over 10 years of big data experience with in-depth knowledge about streaming, batch and ad-hoc data processing. They will not only bring insights on technical directions, but also provide valuable user scenarios which can help us shape the future of openLooKeng. Please join me in welcoming Xulifeng and Xutianli. Best regards, /** * Ken Zhang * openLooKeng PMC */

4 3

转发: Welcome to the "Dev" mailing list
by Jiangdayong 15 Jul '20

15 Jul '20

江大勇 -----邮件原件----- 发件人: dev-request(a)openlookeng.io [mailto:dev-request@openlookeng.io] 发送时间: 2020年7月15日 11:37 收件人: Jiangdayong <jiangdayong(a)huawei.com> 主题: Welcome to the "Dev" mailing list Welcome to the "Dev" mailing list! To post to this list, send your email to: dev(a)openlookeng.io You can unsubscribe or make adjustments to your options via email by sending a message to: dev-request(a)openlookeng.io with the word 'help' in the subject or body (don't include the quotes), and you will get back a message with instructions. You will need your password to change your options, but for security purposes, this password is not included here. If you have forgotten your password you will need to reset it via the web UI.

1 0

Carbondata Auto-Cleanup
by Sandeep K 15 Jul '20

15 Jul '20

Hello all, Openlookeng integration with Carbondata not supporting auto cleanup of vacuum/compaction tables. This becomes the limitation of openlookeng since other processing engine like spark provides the same. Find initial design doc in attachments. Kindly let me know if there are any comments/suggestions. Thanks, Sandeep.k

1 0

Compaction on Carbondata Connector [Carbondata]
by Siddharth Jaiswal 14 Jul '20

14 Jul '20

Hello All, We propose to include Compaction operation for Carbondata data source such that multiple segments of Carbondata files in memory are merged into smaller number of segments, either based on size or count constraints as supplied by the user. The Compaction operation is valid for both partitioned tables and simple non-partitioned tables. We call this operation as VACUUM and have as follows- 1. MINOR Vacuum Operation - based on count of segments to be merged 2. Major Vacuum Operation - based on size of segments to be merged This operation will be beneficial as it will compact the multiple segments that are maintained into a single segment in memory. Attached is the initial design document. Please let me know if there are any comments/suggestions. Thanks and Regards, Siddharth D Jaiswal

1 0

2024

2023

2022

2021

2020

Dev July 2020