Jump to content
GreenSock

martya

Banner development Continuous Integration and Deploy

Warning: Please note

This thread was started before GSAP 3 was released. Some information, especially the syntax, may be out of date for GSAP 3. Please see the GSAP 3 migration guide and release notes for more information about how to update the code to GSAP 3's syntax. 

Recommended Posts

I've put together a CI flow that I've tested on a recent campaign and it's gone pretty well but there's definitely room for improvement. Here's a rundown on how set our workflow up. I'm interested to hear if you guys are doing anything similar or have any opinions on the process.

 
Development
- A campaign has one source control repository.
- Each campaign has a campaign.json file in the root which contains some basic metadata about the campaign.
- Each banner lives in a subfolder within the repo with it's own package.json and gulpfile.js which is generated by the yeoman generator (buildabanner).
- Each banner has a banner.json which contains some basic metadata about the banner (ideally this would be created by the template workflow).

 
 
Continuous Integration
At our agency we primarily develop in the .net framework so I use AppVeyor as our CI tool. As soon as a dev commits, my custom CI script (YAML) loops through each subfolder and runs the gulp build task for that banner, then adds a new folder to a zip file for that banner, containing the distribution files, and the delivery files (zip and backup.gif). We end up with one zip to represent the campaign which is pushed up as the deploy artifact. 
 
appveyor.yml

install:
  - npm install -g gulp

shallow_clone: true

clone_folder: c:\projects\myproject

test: off  

environment:
  campaign_folder: mycampaign
  campaign_zip: mycampaign-banners.zip
  campaign_zip_path: tba

before_build:
  - cmd: set campaign_zip_path=c:\projects\myproject\%campaign_zip%
  - node --version
  - npm --version
  - echo Gulp Version
  - gulp -v
  - echo %CD%
  - cmd: dir /b

build_script:
  #note: 7z can't set the desination path when adding a file so we add the file first, then rename it to move it to the desired folder.
  - cmd: 7z a %campaign_zip_path% campaign.json
  - cmd: 7z rn %campaign_zip_path% campaign.json %campaign_folder%\campaign.json 
  #call batch file - appveyor doesnt support multiline cmds correctly - https://stackoverflow.com/questions/37627248/how-to-split-a-command-over-multiple-lines-in-appveyor-yml
  - cmd: package-subfolders.bat

skip_tags: true

after_build:
  - echo Pushing Artifacts...
  - ps: Push-AppveyorArtifact $env:campaign_zip -DeploymentName bannersdist

deploy:
- provider: Environment
  name: uat

after_deploy:
 - cmd: echo Preview URL will be - http://banners.xxxxx.com/campaign/%campaign_folder%

package-subfolders.bat

FOR /D %%G in ("*") DO (
  Pushd %%G
  Echo Packaging banner - %%G
  npm install
  gulp ba
  7z a %campaign_zip_path% banner.json 
  7z rn %campaign_zip_path% banner.json %campaign_folder%\%%G\banner.json
  7z a %campaign_zip_path% dist\* 
  7z rn %campaign_zip_path% dist %campaign_folder%\%%G\preview
  7z a %campaign_zip_path% delivery\* 
  7z rn %campaign_zip_path% delivery %campaign_folder%\%%G\delivery
Popd )

 
 
 
Continuous Deploy
The deploy artifact from the previous step is unzipped to a specific folder that matches the campaign name on our uat server. 


Preview Tool
I created a really simple .net webapp that sits on the UAT server that reads the folder structure from the banner deploy folder. That's where the campaign.json and banner.json files come in, they provide the extra information to make the preview tool presentable to the internal team/media agency/client. They can all view/download the latest files as they please without having to bother development. Note that I added a "published" flag to the banner.json file so that we can hide banners from appearing on the site until we're ready.

That way we can give out a structured URL - http://banners.xxxxx.com/campaign/[campaign_folder] and you'll see a table with all the banners that are up on the uat server, you can click in and see the embedded banner and backup gif with buttons to download the delivery files etc. I'm also doing a couple of handy things like calculating the total file size of the banner (adding up all the files in dist folder) so we can keep an eye on the sizes at all times.

A few hours of my spare time to build it but it's already paying off, as with the nature of ad agencies we're doing constant revisions under high pressure so it's nice to know the dev can make a change and that the latest versions of the banners will be up on the server at all times (a couple of minutes after the commit) without any manual work. Looking at the appveyor logs, in the space of this one campaign (around 20 individual banners) over the last week the tool has done over 100 builds/deploys. That's hours of manual deploys saved and freeing us up to get on with the job.

 

 

 

 

In the long term I'd like to turn the web app into a proper tool with proper logins and notifications when deploys have been made etc. Are there any products out there that do the job of this preview tool? I've had a brief look before I created it and couldn't really find anything. 

 

 

 

It's not a perfect process, there's heaps I'd like to improve on but it's miles ahead of the old process which was - developer creates working files on his desktop and emails zip files to producer on demand, eventually source files are lost forever.



 

  • Like 3
Link to comment
Share on other sites

If I were to tell you that 99% of the agencies I work with still have the "create working files on the desktop, email zips to producer on demand, save source in some corner of some server", would that make your insides warm and fuzzy?

 

I wish more places would build a pipeline but that doesn't seem to be the place. As far as I know, there is no such de-facto preview system out in the wild. There are the preview and notification systems of each AdPlatform.

 

From my experiences working with different agencies, each have their own way of doing things - from the animation structure to the preview system. Heck, I even came across an agency that had a semi-permanent freelancer who would animate his banners in After Effects, export it as a video and upload the video as the banner.

 

So, you're doing great and on the right path.

  • Like 3
Link to comment
Share on other sites

@martya Thanks for this!

 

It's great to hear how other people are handling banner development at other agencies, and how they're using existing WebDev technology to make things easier and more seamless.

 

I currently use an NPM package called https://www.npmjs.com/package/generator-bannertime which I co-author, we played around with a few deployment techniques.

 

Using git hooks https://medium.com/@joe_midi/git-hooks-and-shell-scripts-da982456d6d8, with this we needed to create a bare repository on the server and use that as a remote, so we could push our files to that point and it will change the working directory.

 

The second method, which I don't like as much because it clutters the Terminal was good ol' fashioned SCP. Simply upload all the public data over SCP, what made this really simple was SSH keys, giving automatic authentication to the terminal. So all I needed to run is gulp deploy and voila, it will push all that data up.

 

Another part of the shell script we created also allowed us to create branches, and form new preview pages for that branch, just in case a client asked: "oh can we have a new version with that there, and this other there?". Or in some region specific cases where the client didn't want to have each country on the same preview page, we would create a new branch for that specific country, and that would be a suffix to the preview page. Such as http://agency.com/banners/client-campaign-country that way we could keep all the files for that one campaign in the same repo, and just keep spinning off new countries as and when the client required them.

 

As we were using BitBucket as our source, I've been looking at BitBucket Pipelines for CI, all it needs to do is have a docker file that has npm, gulp, ssh, git, I can pass credentials to it through environment variables, then I can use that to run gulp deploy when I push to a deploy branch.

 

I'm currently freelancing at an agency that does not have their own preview servers and completely rely on ad platforms, which I find to be a really slow process as you have to do it per ZIP file rather than a batch process or a pipelined task.

  • Like 2
Link to comment
Share on other sites

There was a guy building something like this a year ago or so, was in Beta at the time. A whole preview thing specifically for banners, almost identical to what you're thinking but it was pretty much fleshed out. I was giving him feedback on his app but haven't seen an update since. Was called Banner Monkey or something like that (you may have to Google it).

Like Dipscom said, most agencies keep it simple. Build it wherever you want, zip it up and send it over. 

Personally, I think a Preview Tool is great. I'd pay for something like that.

Thoughts on batch processing / continuous deploy / integration:
I like the idea of batch processing the banner at the end (like minifying and packaging it up). I don't think I'd personally use any kind of system during creative development though, feels too over-complicated.

Not everyone creates banners the same either. Would your app also factor in banners being created with AnimateCC? Would it need to adhere to some kind of specific file structure if it's hand-coded? 

However, if you were to create a way to make dynamic / programmatic banners much easier to create, then that is something I would think agencies would find super valuable. Something like GSAP components or even AnimateCC drag and drop components.

Link to comment
Share on other sites

 

 

"I'm currently freelancing at an agency that does not have their own preview servers and completely rely on ad platforms, which I find to be a really slow process as you have to do it per ZIP file rather than a batch process or a pipelined task."

 

Regarding that  - one tip is that Dropbox by self is a great host (and share) or https://www.kissr.com. 

hth

Link to comment
Share on other sites

Thanks for the input guys. Yeh from the preview tool perspective, it should be possible to build a common display platform because at that point we should be working with the final deploy files only, not the working files, the preview tool shouldn't know or care what the banner was built with, only what the target platforms are (DCM, DCRM etc). 

 

How you build the preview files depends. It could be an export from Animate CC or a build task from gulp/grunt, but the final output of this should match the platform requirements. 

 

CI/CD is an optional extra. If I was building the preview tool as a proper product instead of my CI process publishing to a folder on the server I could instead build an API and have the deploy task push the banner files to that through powershell etc

  • Like 1
Link to comment
Share on other sites

@davit, did you mean this one: https://adpiler.com

I've got an invite for the demo, but I haven't had the chance to take it up.

 

@martya, working in a bigger agency with a lot of different people with different skill sets, a preview tool that could support multiple methods of banner creation would be more useful. I would also be interested in creating automated test scripts for banners, make sure they click through or have the CORRECT click through for whatever platform they are created for, should it be Enabler.exit(), ExitApi.exit(), EB.clickthrough(), etc

 

Much like the DCM/AdWords testers on AppSpot, but that could handle any banner or platform.

  • Like 1
Link to comment
Share on other sites

@martya, working in a bigger agency with a lot of different people with different skill sets, a preview tool that could support multiple methods of banner creation would be more useful. I would also be interested in creating automated test scripts for banners, make sure they click through or have the CORRECT click through for whatever platform they are created for, should it be Enabler.exit(), ExitApi.exit(), EB.clickthrough(), etc

 

Much like the DCM/AdWords testers on AppSpot, but that could handle any banner or platform.

I was thinking exactly the same thing, the only real question is where would the testing take place. Technically if your automating builds, it should be performing the validation in the test phase and fail the build if it's not correct, preventing the banners from being deployed (to preview tool etc).

 

However from a product point of view it makes sense to have some intelligence in the preview tool, I can't see the masses being too keen on running banners through CI. For argument sake the banners could first go into a draft mode on the preview tool which allows for validation to take place before allowing it to be published for clients to see etc.

 

I was thinking last night maybe banner validation could be performed using Selenium tests, I might have a play around with that.

Link to comment
Share on other sites

I was also thinking last night that it would be great to have publisher profiles that you could also test against, and maybe combine with platform specs, ensuring that each build is also correct against publisher as well as platform.

  • Like 1
Link to comment
Share on other sites

  • 9 months later...
On 9/16/2016 at 5:55 AM, joe_midi said:

Using git hooks https://medium.com/@joe_midi/git-hooks-and-shell-scripts-da982456d6d8, with this we needed to create a bare repository on the server and use that as a remote, so we could push our files to that point and it will change the working directory.

 

The second method, which I don't like as much because it clutters the Terminal was good ol' fashioned SCP. Simply upload all the public data over SCP, what made this really simple was SSH keys, giving automatic authentication to the terminal. So all I needed to run is gulp deploy and voila, it will push all that data up.

 

Another part of the shell script we created also allowed us to create branches, and form new preview pages for that branch, just in case a client asked: "oh can we have a new version with that there, and this other there?". Or in some region specific cases where the client didn't want to have each country on the same preview page, we would create a new branch for that specific country, and that would be a suffix to the preview page. Such as http://agency.com/banners/client-campaign-country that way we could keep all the files for that one campaign in the same repo, and just keep spinning off new countries as and when the client required them.

 

As we were using BitBucket as our source, I've been looking at BitBucket Pipelines for CI, all it needs to do is have a docker file that has npm, gulp, ssh, git, I can pass credentials to it through environment variables, then I can use that to run gulp deploy when I push to a deploy branch.

 

Hey @joe_midi, I've been doing some research into git hooks, and of course your medium post about it comes up as one of the prominent results. I was wondering about your setup - we generally create a new git repo for each campaign as well; are you pushing from each campaign to a single bare repo on the server? If I understand bare repos correctly, there's no working tree, so there's no issue with pushing from multiple repos > a single repo, correct? Thanks for the clarification!

Link to comment
Share on other sites

Hey @flysi3000

 

I've never setup a multi -> single git repo up, but I imagine that it could work with some fancy shell scripting.

 

Regarding the working tree, I think you're right by default there isn't one, but how I was using the post-receive hooks I defined one.

 

#!/bin/bash
while read oldrev newrev ref
do
	if [[ $ref =~ .*/deploy$ ]];
		then
		echo "Deploy ref received. Deploying branch to preview..."
		git --work-tree=/var/www/html/project-name --git-dir=/var/repositories/project-name/git checkout -f deploy
	else
		echo "Ref $ref successfully received. Doing nothing: no work-tree with that branch name found on this server."
	fi
done

 

So in my example from that medium post, I have defined my working tree as website.com/project-name, but the git repository is actually stored elsewhere on the server which shouldn't be accessible by the public.

 

I guess you could create a bare repo point, that takes in your ref and branch and figures out where to create that working tree and git dir.

  • Like 1
Link to comment
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
  • Recently Browsing   0 members

    • No registered users viewing this page.
×