- Dev - mailweb.openlookeng.io

Re: Contribute kube-LooKeng to the community
by Shudi Gao 04 Nov '20

04 Nov '20

Hi Kevin, We appreciate your interest in contributing to the openLooKeng project. Deploying in Kubernetes is an interesting usage scenario, and I'm glad to see you are looking into it. I just want to point out a couple of items in the current repo, to make sure we present a consistent Docker and Kubernetes story. Under the "docker" folder<https://gitee.com/openlookeng/hetu-core/tree/master/docker> of the hetu-core repo, there exist scripts and Dockerfile to build an openLooKeng docker image. Also under the "hetu-samples/kubernetes" folder<https://gitee.com/openlookeng/hetu-core/tree/master/hetu-samples/kubernetes>, there is a sample deployment file for Kubernetes. You may want to consider these in your work for optimal reuse and outcome. The above Kubernetes sample uses "hostPath" for configuration files. I did some work to change that to config-maps, for better flexibility. I also have a Helm chart. I hope to contribute them in the near future. I'll be happy to work with you on these. Look forward to your contribution! Regards, Sandy Gao

2 1

Contribute kube-LooKeng to the community
by Zheng Zhenyu 04 Nov '20

04 Nov '20

Hi openLooKeng, I'm a rookie in the openLooKeng community, recently, openLooKeng community provided a very good script to install openLooKeng services. But recently, we have some project that requires me to deploy openLooKeng on K8S clusters, so I referenced docker and k8s related resources in the current codebase and started a kube-LooKeng project in my github repo. Currently, I've got some achievements, users can now create openLooKeng clusters with a "Airpal" frontend using my script. The repo is here: https://github.com/ZhengZhenyu/kube-LooKeng [https://avatars3.githubusercontent.com/u/10849016?s=400&v=4]<https://github.com/ZhengZhenyu/kube-LooKeng> ZhengZhenyu/kube-LooKeng<https://github.com/ZhengZhenyu/kube-LooKeng> Deploy openLooKeng cluster using Kubernetes. Contribute to ZhengZhenyu/kube-LooKeng development by creating an account on GitHub. github.com Even though the project is still at its begining stage, and highly hardcoded to our own case now, I would like to propose to open source it to the openLooKeng community, so that people that are also interested could join to work together. And I also have future plans like enable "Kustomize" and/or "Helm" to make it more configurable. BR, Kevin Zheng

3 3

Merging small file size splits Design Doc
by K Sandeep 28 Oct '20

28 Oct '20

Hi all, For hive datasource, driver is created per split, so more splits means more parallelism, this is working without any issue if we have good file size , however , there is a case which the file size is far less than hive split size(hive.max-initial-split-size, hive.max-split-size), thus number of splits is determined by number of small files. To process a large number of small splits in parallel, it definitely introduces overhead of CPU context switch, causing performance issues when doing the data shuffling and resulting in low CPU/memory usage. Based on our testing we are seeing huge performance impact in high concurrency environment since task slot is occupied by those small split. Find design doc in attachments. Kindly let me know if there are any comments/suggestions.

1 0

Merging small file size splits Design Doc
by K Sandeep 09 Oct '20

09 Oct '20

Hi all, For hive datasource, driver is created per split, so more splits means more parallelism, this is working without any issue if we have good file size , however , there is a case which the file size is far less than hive split size(hive.max-initial-split-size, hive.max-split-size), thus number of splits is determined by number of small files. To process a large number of small splits in parallel, it definitely introduces overhead of CPU context switch, causing performance issues when doing the data shuffling and resulting in low CPU/memory usage. Based on our testing we are seeing huge performance impact in high concurrency environment since task slot is occupied by those small split Find design doc in attachments. Kindly let me know if there are any comments/suggestions. Thanks, Sandeep.k

1 0

openLooKeng PMC Biweekly Meeting
by Maowei (Morewell) 12 Aug '20

12 Aug '20

openLooKeng PMC Biweekly Meeting Agenda: * Review open issues from previous meetings o Find ways to expand openLooKeng in Canada and in India（Since it’s close to back-to-school season, how can we setup meetup groups in Toronto & India）――Ella Liu o Vote on the features to determine the priorities ――Michael * Push down design review――Luodan * Document management tool discussion――Yujiwen If you have other topics to discuss please leave a message here：https://etherpad.openlookeng.io/p/PMC-weeklymeeting Meeting Link: https://welink-meeting.zoom.us/j/315761172

1 0

openLooKeng PMC Biweekly Meeting
by Maowei (Morewell) 31 Jul '20

31 Jul '20

Just remind that our PMC Biweekly meeting will be held at 11 a.m. Beijing time today~ openLooKeng PMC Biweekly Meeting Agenda: * Remaining Issues of last meeting * Review the features of 930 vesion ――Michael * Introduce PMC's voting rules and methods for requirements and issues――Fred Li If you have other topics to discuss please leave a message here：https://etherpad.openlookeng.io/p/PMC-weeklymeeting Meeting Link: https://welink-meeting.zoom.us/j/663235710

2 1

Remove he segment when all segments are deleted in a segment.
by Neeraj Unnikrishnan 30 Jul '20

30 Jul '20

Hello all, This proposal is to add an enhancement to the carbondata connector which marks the entire segment as marked for delete if all the elements in that segment is deleted. Please have a look and let me know if you have any suggestions or comments. Thanks & Regards, Neeraj U

1 0

openLooKeng PMC Biweekly Meeting
by Maowei (Morewell) 29 Jul '20

29 Jul '20

openLooKeng PMC Biweekly Meeting Agenda: * Remaining Issues of last meeting * Review the features of 930 vesion ――Michael * …… If you have other topics to discuss please leave a message here：https://etherpad.openlookeng.io/p/PMC-weeklymeeting Meeting Link: https://welink-meeting.zoom.us/j/663235710

1 0

Auto-Compaction
by K Sandeep 28 Jul '20

28 Jul '20

Hi all, Openlookeng not support auto compaction of tables. This becomes the limitation of openlookeng since other processing engines like spark provide the same. Find design doc in attachments. Kindly let me know if there are any comments/suggestions. Thanks, Sandeep.k

1 0

Introduce the new work mechanism for approvers and reviewers
by zengchen 28 Jul '20

28 Jul '20

Hi all contributors: My name is zeng Chen and in charge of the service of robot for openlookeng. Recently, some contributors ask me to manage the approvers and reviewers as this one which is already running on k8s. I think it is a good suggestion. I send this email to clarify how does that mechanism run. 1. How to manage approvers The approvers are managed though the OWNERS files which may exist at each level directories. An approver in an OWNERS file can approve(by adding approved label) any file in that directory and its sub-directories, but he/she can't approve the files in the parent directory and any other sibling directories. 2. How to add label of 'approved' Please see the example to get the details. 3. How to manage reviewers Like the way to manage the approvers, reviewers are managed by OWNERS file too. A reviewer in an OWNERS file can approve(by adding lgtm label) any file in that directory and its sub-directories, but he/she can't approve the files in the parent directory and any other sibling directories. This is the big change to before. The old mechanism is that any one of collaborators which are managed by the git platform such as Gitee, Github can add lgtm label. The purpose is to let the reviewer work as approver and make each files at different level directories reviewed by different reviewers. 4. How to add label 'lgtm' The process is also like approve process, see the example of adding approved label above. In summary, both the approver and reviewer will be managed by OWNERS file and their working mechanisms are also the same. Appreciated for any suggestions. Thanks! Best Wishes zeng chen

2 2

2024

2023

2022

2021

2020

Dev