- Dev - mailweb.openlookeng.io

[doc]new requirement about guidance to tryme
by Liyongle (Fred) 22 Jul '20

22 Jul '20

Hi docs team, I created [1] for the guidance to tryme. Please find the details in the issue, and leave comments there or by replying this email. Thanks [1] https://gitee.com/openlookeng/hetu-core/issues/I1OQ2E Fred 李永乐

1 0

MOM- openLooKeng PMC Biweekly Meeting at 11:00 a.m. Beijing time on July 17
by Maowei (Morewell) 21 Jul '20

21 Jul '20

Biweekly meeting at 11:00 a.m. Beijing time on July 17. +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Attendees list: �� Ken Zhang �� Michael Li �� Raghunandan �� Xulifeng �� Xutianli �� Zhangwei �� Yangguo �� Maowei �� Zhangheng �� Fred Li �� Huangyan �� Ella Liu �� Lizheng �� Nitin �� Xudezhi �� Liaodenghong �� Rajeev rastogi �� Zhangjingfang Agenda: �� PMC Operation Mechanism �� Ken �� Vision of openLooKeng��Michael �� 930 openLooKeng version features ��Michael �� Proposed openLooKeng RoadMap ��Michael Minutes: �� PMC Operation Mechanism * Major responsibilities of PMC �� Final technical decision-making authority of the open source code base �� Open is mandatory, all communication must be visible �� Maintain roadmap and milestones �C Ken, Xulifeng �� Manage efficiency, quality and dependency of development process �C Raghu, Chenzeng �� Design reviews �C Michael �� Maintain approver/committer lists - All �� Community expansion �C Michael, Maowei, Raghu * Roadmap & Milestone-Ken��Xulifeng �� Vision - responsible for defining the system positioning and technology direction �� Roadmap and milestone for the second half year of 2020 �C 1st draft on July 30th �� Quarterly releases �C 1st official release: 930 �� Major semi-annual releases �� Release Manager �C Maowei * Software Development Life Cycle (SDLC) efficiency and quality��Raghu �� Responsible for the proper SDLC process to support quality development �� Design guidelines, Coding guidelines, UT �� Code inspection * Design reviews��Michael �� All design review requests to be raised on slack and dev(a)openlooken.io<mailto:dev@openlooken.io> for visibility �� Ensure in-depth discussion on all features �� All features must have design docs in our community �� Only approved feature with proper design guidelines can be merged * Community Expansion �C Ella (Canada), Maowei (China), Raghu (India) �� Community expansion events will be organized independently in each location while some materials will be shared �� Collect a list of all communities can help our expansion �� University Events �� Hackthon * PMC Activity Organizers �C Maowei/Ella �� Biweekly meetings �� Each meeting must have an agenda sent out ahead of time �� Meeting minutes must be sent to dev/user mailing list after finished �� Vision of openLooKeng * A stupid and simple in-situ engine to satisfy ad-hoc, batch, and streaming use cases * Plug and Play capability to improve performance * Global data access with simplified view via virtualization capability �� 930 openLooKeng version features * Further discussion is needed. In the next PMC meeting we should review items with priority for 930 release �� Proposed openLooKeng RoadMap * 2020 Q4��Focus on engine enhancements which includes core, performance and north-south enhancements. * 2021 Q1��Focus on Cross DC Performance Improvements��CBO enhancements and integration with hardware * 2021 Q2��Focus on Stream processing��Batch processing etc. Remaining Issues�� Collect a list of communities that can help us to expand��Fred Li / Maowei �� Where do we archive the design docs��Michael �� Find a tool that can be used for voting features on an open source platform��Fred Li Also you can see this MOM in here��https://etherpad.openlookeng.io/p/PMC-weeklymeeting +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

1 0

Alter Table Support For Carbondata Connector
by Neeraj Unnikrishnan 20 Jul '20

20 Jul '20

Hi all, This proposal is for providing the support for Alter table operations on Carbondata connector. Attached is the design document for the same. Please let me know if you have any comments or suggestions.

1 0

Table Pushdown Query Optimizer
by Siddharth Jaiswal 20 Jul '20

20 Jul '20

Hello All, We propose to include a query optimizer called Table Pushdown which is a rule based optimizer and acts upon queries which satisfy the constraint that the query has a subquery within it and the outer table must have a join with the inner table specified in the subquery on a unique/Primary Key column of the outer table. The following reference might be helpful- https://15721.courses.cs.cmu.edu/spring2017/papers/15-optimizer2/a8-sen.pdf The expectation is that by using this query optimizer, the run time can be reduced significantly for such queries thereby providing a significant advantage to the OpenLookeng engine's query processing capabilities. Attached is the design document. Please let me know if there are any review comments/suggestions. Thanks and Regards, Siddharth D Jaiswal

1 0

Auto-Compaction for Carbondata Connector
by K Sandeep 20 Jul '20

20 Jul '20

Openlookeng integration with Carbondata not supporting auto compaction of tables. This becomes the limitation of openlookeng since other processing engine like spark provides the same. Find initial design doc in attachments. Kindly let me know if there are any comments/suggestions. Thanks, Sandeep.k

1 0

[all]PPT templates are available
by Fred Li 17 Jul '20

17 Jul '20

Hi community, Thanks to lvzixin who helped us to design the ppt templates[1]. [1] https://gitee.com/openlookeng/community/tree/master/artwork/ppt_template -- Regards Fred Li (李永乐)

1 0

openLooKeng PMC Biweekly Meeting
by Maowei (Morewell) 16 Jul '20

16 Jul '20

openLooKeng PMC Biweekly Meeting Agenda: * PMC Operation Mechanism * Large-granularity Milesetone & RoadMap * 930 openLooKeng version features * Other…… If you have other topics to discuss please leave a message here：https://etherpad.openlookeng.io/p/PMC-weeklymeeting Meeting Link: https://welink-meeting.zoom.us/j/813957346

1 0

Auto cleanup for Hive ORC compaction
by Aman Omer 15 Jul '20

15 Jul '20

Hi all, Since, openlookeng is now supporting compaction/vacuum operation on Hive transactional tables which merges all base and delta files together (according to the compaction type). So that upcoming read queries can be redirected to use newly created, compacted directories which improves performance. This makes old, smaller delta directories eligible for cleanup. Therefore, a background process is required which will be responsible for removing any redundant directory from table path. Check attached document for more details. Kindly let me know if there are any comments/suggestions. I have created a slack channel (#auto-cleanup). Kindly join it for discussion. Thanks, Aman Omer

1 0

Carbondata Auto-Cleanup
by K Sandeep 15 Jul '20

15 Jul '20

Hello all, Openlookeng integration with Carbondata not supporting auto cleanup of vacuum/compaction tables. This becomes the limitation of openlookeng since other processing engine like spark provides the same. Find initial design doc in attachments. Kindly let me know if there are any comments/suggestions. Thanks, Sandeep.k

1 0

Carbondata Auto-Cleanup
by Sandeep K 15 Jul '20

15 Jul '20

Hello all, Openlookeng integration with Carbondata not supporting auto cleanup of vacuum/compaction tables. This becomes the limitation of openlookeng since other processing engine like spark provides the same. Find initial design doc in attachments. Kindly let me know if there are any comments/suggestions. Thanks, Sandeep.k

1 0

2024

2023

2022

2021

2020

Dev