ZenPack Development : ZenPack Development Process : Comments

RE: ZenPack Development Process

community@zenoss.org — Thu, 22 Sep 2011 15:28:24 GMT

Note: This comment and the following 3 comments are refering to a much older version of the above document. The old document was drastically different and breif.

While it's good to have a concise set of instructions, this page really needs to be expanded. For some of these steps it is not clear in which direction the updates are occuring. It would be better to stay away from terms like 'sync with' in favor of 'sync from/to'. It would also be good to understand which steps are benign and which could clobber your project. And give a few checkpoints along the way to confirm the steps performed so far are correct.

While these steps are helpful in the case of editing files directly from the current shell session, it's not clear how changes made from other sources are merged in. If I use an external editor/IDE and then copy in new files, I assume they need to be manually added and committed. But what about changes made from Zenoss? If I export the zenpack for example, and zenoss updates my objects.xml, does that need to be manually committed? If so, to which branch: my master or my development branch?

Perhaps it is just my inexperience with git but this seems like a lot of trouble for developing just a zenpack. If I were working collaboratively or making changes to other people's zenpacks (which I don't even think is possible here) I can see why this would be necessary. But for a single developer working on a single zenpack: is it really necessary to push, pull and merge the entire Zenpack tree?

RE: ZenPack Development Process

community@zenoss.org — Fri, 23 Sep 2011 06:58:24 GMT

Dave, thanks for the feedback. These instructions were made a while back, when we were first moving to Github. They are in need of updating.

We have an active plan and project to improve Community ZenPack SCM practices. Work is planned for A) the layout of zenpacks on github B) clarification of our suggested dev workflow and C) our dev and github documentation. Simon Jakesch and I are heading this up and we will plan to evaluate and incorporate where-possible, your suggestions.

A) Layout

We are going to change the structure of our ZenPacks on github. Likely, it will look like this: Each ZenPack will have its own seperate repo. No more cloning the entire tree of ZenPacks. Additionally, we may be moving Zenoss Core development over to github too.

B) Workflow

Not sure of all what is in store here yet. First, read this superb community-written document in detail: http://community.zenoss.org/docs/DOC-10223 (download the PDF). It is a must-read. It documents the mechanism / workflow for developing Zenoss ZenPacks. It may answer many of your questions.

C) Documentation

Here are some more of what I want to look into for docs around this:

An official and reviewed document that goes over the mechanics of the ZenPack development process. Right now, it is this: http://community.zenoss.org/docs/DOC-10223
I think that there should be a concise / cheat-sheet document, accompanied by additional documents that dive into more detail.
I am also thinking that there should be different tutorial use-cases such as:
- I am totally new to developing with Zenoss, how can I get started
- I already dev'ed my ZP, how do I put it on github?
- I want to make some changes on someone else's public ZenPack.
- I know how to dev ZPs, but Ive never done git / github. I am still dev'ing a ZP, what do I do?
- I have developed a ZP entirely in the Zenoss GUI, how do I setup and submit this to github?
- Git / Github / Zenoss suggested workflow and environment

Answers to your questions:

Many of your questions surround around the development mechanisms explained in http://community.zenoss.org/docs/DOC-10223

If I use an external editor/IDE and then copy in new files, I assume they need to be manually added and committed.
- I am thinking that you may have not implemented step "3.3.Relocate the ZenPack to the local repository" of the above doc.
- Once you have, any edits with an IDE are also now to the git repository. Either as soon as you add any files, or once you get to a commit point (dont wait too long between commits), follow the directions below.
How do I add and commit files locally?
- A: run "git add *", then run "git status" and finally "git commit -m 'new code that will introduce select-all functionality'"
  - this assumes that you are in the local repo directory
  - "git add *" will stage (prepare) all files in the current and child directories to be ready for commit. It does not actually change any files or make any commit action. It just stages the changes, so that you can know for sure what you are committing. You could also run "git add README" or "git add \*.log" to do some variety of files. You will need to run this on all changed files everytime you want to commit, even if you have added the same files in the past. There is a shortcut to avoid using git add altogether (git commit -a -m 'blah'), but know what you are doing before using it.
  - "git status" shows what is staged by the prior command. Helps you check your work to make sure that you have staged everything that you want.
  - "git commit -m" snapshots the changes shown in git status to the current branch(?). The -m is a required comment on what the commit did. Make this short and descriptive to what the change implements or does functionally. Kind of like the comments you put on wikipedia changes.
If I export a zenpack and files change, what do I do?
- A: Again, if you have implemented step "3.3.Relocate the ZenPack to the local repository" of the above doc, there should be no problems here because any changes that the export makes, is a change to the files in the local repository. You could run "git add /ZenPacks/community/IBMSystemxIMM/objects/objects.xml" ("git add *" is easier) and then "git commit"
To which branch?
- I am not so clear yet on branch use-case. I would venture to think that you are making these all on a seperate development branch. You do everything on a dev branch, until you have tested, installed, uninstalled, and know that your work is ready for prime-time. This is, at least, what is depicted by a few of our docs.

General Comments on using Git

It may seem a little difficult to use github for one-man ZenPack work, but our hope is that more collaboration takes place, that we have one flexible platform for all our code, that others can easily fork and allow access to their code (even without going through us), and to automate our workflow when getting new submissions (no more backlog). Additionally, I look at it as a personal investment. It will be good to be a master at git, so you can use it elsewhere.

RE: ZenPack Development Process

community@zenoss.org — Fri, 23 Sep 2011 07:11:30 GMT

Answers to other common questions:

How can I pull down changes made to the main github repo (github.com/zenoss/Community-ZenPacks) ? / How can I sync from the official ZenPack repository?
- A: all you need to run is: "git fetch upstream" and "git merge upstream/master"
  - "fetch" gets the data from upstream (github) and saves it in a temporary place locally - it is not merged in yet.
  - "merge" actually brings this new data that you got from fetch and merges it with the main local branch (master). Now you have the latest stuff from main github.
  - This assumes that the definition for your upstream remote (github) is in place (check by running "git remote -v". If its not defined, or you are not sure, use: "git remote add upstream git://github.com/zenoss/Community-Zenpacks.git" )
How can I push my local changes to my own github repo (ex: nyeates/Community-ZenPacks)?
- A: You first had to have created a github repo (done in the github web gui while logged in, button called 'New repository'). You ussually name it the same as the repo that you already have locally (ex: testSplit).
- Next, you define this new remote repo source. While in the local repo dir, run "git remote add origin git@github.com:nyeates/testSplit.git". Replace with your username and repo name.
- Finally, push your local repo to the github remote repo with "git push -u origin master". This pushes to origin (github) from the master branch (local).

RE: ZenPack Development Process

community@zenoss.org — Tue, 27 Sep 2011 18:54:41 GMT

Thanks Nick, this is quite helpful and answers some of my questions. I did also read David B's document (mentioned above) and found it very helpful for the github stuff specifically. But it did not have more detail on the actual git operations than what was listed here; that's why I posted the questions.

I should mention that I did relocate the Zenpack to the local repository as describe by David (and also by Jane). I understand why that's a 'best practice' and that part never gave me any problems. I would not have dared sync the Zenpack repository to the Zenoss runtime directory!

Most of my problems were clearly git newbie issues. I understand now that my local git repository treats changes the same way regardless of whether I edit a file locally, copy a file in remotely, or allow Zenoss to update a file, as in the case of objects.xml. And these changes require a 'git add' and 'git commit' prior to merge. Thanks for the tip on 'git status' command. Checkpoints are important for us rookies.

Without recounting every mistake I made (and there were many :- ) I'll focus just on this: Merging to github the first time wasn't so bad, as my Zenpack did not already exist in the tree. But when I updated my Zenpack I was afraid my existing files on github might overwrite my new files when I merged the (github) upstream with my (local) master. And that's exactly what happened. I won't go into detail as it was almost certainly 'pilot error'. But that's why that section of the instruction should be expanded, and a few checkpoints added.

I'm glad to hear you are planning to realign the repositories so individual ZenPack submodules can be developed without having to clone entire ZenPacks tree. That would eliminate the problem I just described. Understood: there are scenarios where the true collaborative environment is required. But for single Zenpack development it would be a welcome simplification. I guess the trick is to align it so you can support both simultaneously: the collaborative environment for advanced users and the individual environment for rookies.

RE: ZenPack Development Process

community@zenoss.org — Wed, 23 Nov 2011 23:06:26 GMT

Nick, Does this imply that the links on the zenpack listing page should link to the github repo(markdown page) rather than to the existing jump pages? Rather than maintaining both (when an existing jump page exists) does it make sense to either delete the existing jump page, or remove its contents and replace with a link to the markdown page. The advantage to the second is that comments wouldnt be lost?

In the past, ZenPack documentation or what were sometimes called "jump pages" were made on the community portal website (Jive; ex: Advanced Device Details). This is now deprecated.

RE: ZenPack Development Process

community@zenoss.org — Fri, 25 Nov 2011 18:25:49 GMT

I kept the post brief on this to keep it simple. This needs thought and discussion. Basically, it is a large task on our list that needs to be done. It would be great to get community involvement on deciding how to best implement the change and help in implementing the change.

For now, maybe keep the jump page with minimal info (initial description and screenshots) (or just keep what is up already) and link off to the github repo for details. As you said, this keeps commenting abilities.

What I would love to do is have the jump pages on jive reflect the content in the README.markdown files. Markdown file would become the master document. This would keep commenting that I do not think github has.

Actually, we could rid of jump pages if github had comments and a rating system. I did not see such things. Let me know if you see different.

Any ideas on how we might ideally set this up and automate it?

Thank!

-Nick

RE: ZenPack Development Process

community@zenoss.org — Sat, 26 Nov 2011 23:53:21 GMT

The jump pages have a few advantages that I can think of:

1) People are used to them.

2) Contents can produce hits when someone searching the community site

3) They allow direct commenting.

I think #2 might be the most important. If the pages were moved to github It would now require users to search 3 sites for information (zenoss.org, the ZCA wiki, and then github). Given that a lot of people don't develop Zenpacks driving those users to Github while searching for answers feels wrong. From my vantage point I think the best compromise is:

1) Making the readme.markdown the "master" document

2) Have some tag or flag in the file that references the jump page.

3) There is an automated routine (or manual script) that would take the markdown, run it through the render, and replace the existing contents of the jump page with the rendered output.

Perhaps part of the process that Zenoss performs when accepting the pull request would be to fire off the script to re-populate the jump page contents?

off-cuff-example... I add the following two tags to my readme file:

<sync_jump_page=True />

<jump_page_url="http://community.zenoss.org/docs/DOC-11352" />

Upon converting/updating those get stripped from the final output. Meanwhile this gives folks the ability to "subscribe" to the process or not.

RE: ZenPack Development Process

community@zenoss.org — Thu, 15 Dec 2011 17:49:51 GMT

Hi Nick,

i want to better understand how sync my existing zenpacks so i need few info:

1- i just read all the docs i found but it's better if i fork my own zenpack or if i create a new version?

2- i try to local copy zenpack but i've issues

[root@ipmonitoring git]# git clone https://andreaconsadori@github.com/andreaconsadori/Community-Zenpacks.git //andreaconsadori@github.com/andreaconsadori/Community-Zenpacks.git

Cloning into Community-Zenpacks...

Password:

error: SSL certificate problem, verify that the CA cert is OK. Details:

error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed while accessing https://andreaconsadori@github.com/andreaconsadori/Community-Zenpacks.git/info/refs //andreaconsadori@github.com/andreaconsadori/Community-Zenpacks.git/info/refs

fatal: HTTP request failed

[root@ipmonitoring git]# git clone https://andreaconsadori@github.com/andreaconsadori/Community-Zenpacks.git //andreaconsadori@github.com/andreaconsadori/Community-Zenpacks.git

Cloning into Community-Zenpacks...

Password:

error: SSL certificate problem, verify that the CA cert is OK. Details:

fatal: HTTP request failed

[root@ipmonitoring git]# git clone andreaconsadori@github.com/andreaconsadori/Community-Zenpacks.git

fatal: repository 'andreaconsadori@github.com/andreaconsadori/Community-Zenpacks.git' does not exist

[root@ipmonitoring git]# git clone git@github.com:andreaconsadori/Community-Zenpacks.git

Cloning into Community-Zenpacks...

The authenticity of host 'github.com (207.97.227.239)' can't be established.

RSA key fingerprint is 16:27:ac:a5:76:28:2d:36:63:1b:56:4d:eb:df:a6:48.

Are you sure you want to continue connecting (yes/no)? yes

Warning: Permanently added 'github.com,207.97.227.239' (RSA) to the list of known hosts.

Permission denied (publickey).

fatal: The remote end hung up unexpectedly

3- i've some zenpacks that i recover from backup and are not in developer mode, what's the best way for this zenpack?

there's a tric or i've to make local a new zenpack and then copy the template ?

regards.

p.s.

i'm not a coder so i've some difficulties with this new github repository methodology.

RE: ZenPack Development Process

community@zenoss.org — Mon, 19 Dec 2011 16:42:05 GMT

Andrea,

Sure, I can help here, appologies for the wait. Let me answer your questions one at a time.

1) You asked if you should fork or make new. Let me give two examples.

If you have updated the existing ZenPack "ZenPacks.AndreaConsadori.AlliedTelsys" which already exists at https://github.com/zenoss/ZenPacks.AndreaConsadori.AlliedTelsys , then you want to fork then clone this one.
If you have a totally new ZenPack that no one has seen before and it is named "ZenPacks.community.CoolNewZenPack", then you want to create your own.

RE: ZenPack Development Process

community@zenoss.org — Mon, 19 Dec 2011 16:50:33 GMT

The errors that you are getting looks like your SSH key is not setup with github correctly. Both your computer and github.com website need to know the same SSH credentials to agree upon. Please assure that you have followed these instructions.

If you have issues even after assuring the above instructions, I found that sometimes my ssh identity became non-active. I fixed this with the following commands "ssh-add -l" to see which ssh identities are active. If none, you can add with "ssh-add ~/.ssh/id_rsa" (or whatever the path to your private RSA identity file).
I see that you are trying to clone "github.com/Community-ZenPacks" repository. This is a deprecated repository. We no longer use it. Do not clone this repository. Also, please read through the commands on this page above. They are very different now.

Instead, each of your ZenPacks are in individual repositories on their own. Browse them at http://github.com/zenoss or in the Submodule repository.

You will first need to fork the repository by clicking "Fork" in the GUI at https://github.com/zenoss/ZenPacks.AndreaConsadori.AlliedTelsys (assure that you are logged in).

After it is forked, it now exists at https://github.com/andreaconsadori/ZenPacks.AndreaConsadori.AlliedTelsys . You can now clone it to your local machine with:

"git clone git@github.com:andreaconsadori/ZenPacks.AndreaConsadori.AlliedTelsys.git"

RE: ZenPack Development Process

community@zenoss.org — Mon, 19 Dec 2011 17:15:40 GMT

3) There is a trick to getting ZenPacks back to Developer mode. See Restoring ZenPacks to Development Mode . You are lilkely to use the second option.

RE: ZenPack Development Process

community@zenoss.org — Wed, 21 Dec 2011 14:20:44 GMT

We suddenly seem to have moved from markdown as a documentation standard to README.rst (having just worked out markdown). Is there a recommended Zenoss template for README.rst?

Thanks,

Jane

RE: ZenPack Development Process

community@zenoss.org — Wed, 21 Dec 2011 18:06:54 GMT

This is true Jane - appologies for jumping around. The background, is that markdown was originally chosen, and then later, Chet did research into various solutions and also Egor spoke up for reStructuredText.

The research and community voice for rst showed that it would meet our needs better. So far, this has been a silent switch. Should it be announced? If so, where might you suggest?

Reference README.rst info can be found here:

RE: ZenPack Development Process

community@zenoss.org — Wed, 21 Dec 2011 18:17:23 GMT

Another question on holding ZenPacks on git.

By default, the .gitignore excludes build and dist so what is the recommended way for providing the egg file?

Cheers,

Jane

RE: ZenPack Development Process

community@zenoss.org — Wed, 21 Dec 2011 21:28:41 GMT

Jane, good question - keep them coming.

See as an example: https://github.com/dpetzel/ZenPacks.community.f5/downloads

Visit one of your own ZenPacks, while logged into github. For example: https://github.com/jcurry/ZenPacks.skills1st.MenuExamples

Now click on the "Downloads" tab, or simply tack on "/downloads" to the url to make it:

https://github.com/jcurry/ZenPacks.skills1st.MenuExamples/downloads

In this area, you can upload your own files - .egg files. And guess what? No more .egg.zip's!

Also given as an option to outside users is the ability to download the code in .zip or .tar.gz formats. Dont confuse this with you still having to upload the compiled .egg's

The .egg that you put in this location can also be referenced in the README.rst as a simple URL. An example of this URL might be: https://github.com/downloads/jcurry/ZenPacks.skills1st.MenuExamples/ZenPacks.skills1st.MenuExamples-1.0-py2.6.egg

So, all said and done, README.rst will supplant jive "jump pages", which will have links to .egg's. "Jump pages" are still planned to exist, and will be copies of the in-repo README.rst's .

RE: ZenPack Development Process

community@zenoss.org — Thu, 22 Dec 2011 12:09:10 GMT

Too complex! Too may options!

We have to make this as simple as possible if people are going to use it.

I have decided to change the .gitignore to include dist and *.egg files. Then, when you add/commit/push it includes the dist directory with the egg file.

In README.rst (which definitely does need documenting, publicising and template providing) you can then provide a link to your dist/<zenpack>.egg file.

Download

========

Download the appropriate package for your Zenoss version from the list

below.

* Zenoss 3.0+ `Latest Package for Python 2.6`_

.. External References Below. Nothing Below This Line Should Be Rendered

.. _Latest Package for Python 2.6: https://github.com/jcurry/ZenPacks.ZenSystems.olsonPower/blob/master/dist/ZenPacks.ZenSystems.olsonPower-1.2-py2.6.egg?raw=true

Note that the directory for dist/<zenpack> seems to need /blob/master/ inserted before dist - more git magic I guess.

Cheers,

Jane

RE: ZenPack Development Process

community@zenoss.org — Thu, 22 Dec 2011 18:02:15 GMT

First to Jane's direct comments, I'd suggest not altering your .gitignore, but rather leverage the downloads section of GitHub. With the approach you've taken, Downloading an older version of the EGG becomes more challenging. I understand that you can go backward intime through the commit history if need be to get an older EGG, but every time you push, the only egg that is displayed is the most up to date. Perhaps this is the desired effect, but as a user I like having the option to pick any version I want. So each time a pack is 'released' that EGG can be uploaded to the downloads section of your Repo and users can make a choice on which version they want, or they simply click your 'latest' link in your file. Again this is my personal opinion and not a "you should do this statement" so take it or leave it.

As for the larger discussion, I think this discussion is much larger than this thread, but some thoughts I've had going through the process.

I don't believe the process has too many options or is too complex for ALL, but that is 100% a true statement for some. Let me try to explain.

Using an SCM system for the first time is complicated at first. I don't care if its GIT, Mercurial, SVN, <insert SCM name here>. The reality of it is it takes some practical hands on for SCMs to make sense. For some folks that develop on a regular basis this is common day, so its second nature to them. For a large percentage of the user base I have to assume they are NOT developing on a daily basis, and as such source control in general is a foriegn concept. Your average System Administrator probably doesn't work with source control much (if at all).

One of the common 'gripes' I've read in the IRC channel and the forums is that "Zenoss Developers/Engineers" need to provide more documentation and best practices. I believe this is exactly what this document is trying to provide, and to that effort I think its a valuable document. If your writing code, putting it into source control is always a good idea, and to that end Github is a decent option. It has a lot of features such as SCM, wiki, issue tracking, and build/release downloads. So for providing an accessible system that 'developers' can use to collaborate and share code this is great step. The learning curve involved with git if your are already familiar with another SCM is fairly low (IMHO).

Now that aside, how many people are "writing code" vs simply creating Monitoring templates using SNMP or perhaps Nagios/Cacti plugins? I don't know the answer to what the percentage is, but the reality is I think its safe to say more than 0 people are contributing ZenPacks that include no custom code. So for someone that doesn't write code, but wants to contribute something back in the form of a monitoring template created in the UI this process is waaaayy overkill.

I think there needs to be some middle ground here. If I'm an SA that hacked up a quick template for an application I support, I should be able to contribute that back without the need to create a GitHub account, learn how to use GIT, learn reStructuredText, etc.

I want to re-iterate I think storing the ZenPack source code on GitHub is a good idea, and I can also appreciate wanting ALL ZenPacks in one place. However, I think there needs to be an avenue for the more casual contributor. Whether that be the ZCA or actual Zenoss Inc resources, someone should be able to hand off their exported egg.

RE: ZenPack Development Process

community@zenoss.org — Wed, 04 Jan 2012 13:25:36 GMT

Why are there two different URLs to a downloads area and which should we use???

You quote https://github.com/downloads/jcurry/ZenPacks.skills1st.MenuExamples/ZenPacks.skills1st.MenuExamples-1.0-py2.6.egg which does work provided you type it all in - note the dowloads before the name in the URL - but this doesn't satisfy the sensible comment from dpetzel about having access to a variety of downloads. To get there, you need a URL like https://github.com/jcurry/ZenPacks.skills1st.MenuExamples/downloads where presumably you can have multiple files for download.

Cheers,

Jane

RE: ZenPack Development Process

community@zenoss.org — Sat, 30 Jun 2012 16:34:27 GMT

Would it make sense to have an extra directory layer so that the material, such as source code, could be held in a subdirectory within the same repository as the Zenpack, but the Zenpack itself is a separate subdirectory?

To me it seems odd that specific source code, as for example for a Flash plugin, is held in a completely different repository.