class: center, middle, inverse, title-slide # Bring your R Application Safely to Production. ## Collaborate, Deploy, Automate. ### <p>Riccardo Porreca, Peter Schmid<br/> (Francesca Vitalini)</p> ### <p>e-Rum2020 Workshop<br/> 2020-06-20 14:00-17:00 CEST <img src="img/mirai_logo_inverted_manual.png" style="width: 450px; position: fixed; bottom: 0; right:0"></p> --- layout: true <div class="my-footer"><span>Copyright (c) 2020</span></div> ---  - Interdisciplinary team of data scientists, software engineers and IT architects - Specialist knowledge ranging from finance, risk management and actuarial knowledge to math / stats techniques - 100+ years of cumulative professional experience in financial services / insurance, software engineering, and professional training ### Bringing ideas to life. ### Smart. Agile. Personal. <img src="./img/mirai-solutions-logo-flyer.png" width="50%" style="position:absolute;bottom:100px;right:50px;" /> --- ## Mirai Solutions - Involved in organizing e-Rum2020 - Eager contributors to open source projects <img src="./img/sticker-pyramid.png" width="50%" style="position:absolute;bottom:80px;right:100px;" /> --- ## Your Miraiers for the day .pull-left[ ### Riccardo - Senior Solutions Consultant - Computer Science / Automatic Control Engineer by education - Academia :: Matlab => Mirai :: R (and more) - R enthusiast since then - Analytics in enterprise environments - Open source projects ] .pull-right[ ### Peter - Solutions Consultant - Environmental scientist by education - Using R since many years, first for statistics, now lots more - Analytics in enterprise environments - Passionate about machine and deep learning ] -- .center[ #### With VanLog's support: ### Andrea Melloncelli ] --- ## Workshop overview .center[ **Workshop repository:** https://github.com/miraisolutions/eRum2020Workshop#readme ] .pull-left[ - Working example: packaged Shiny app on GitHub - Automated checks on Travis CI - **Break** - GitFlow, branch control and pull requests - Agile approach - Setup shinyapps.io for deployment - **Break** - GitHub Actions workflows - Continuous Deployment - Release - More playground ] -- .pull-right[ - (Meant to be) **loosely-coupled** - **Hands-on** - First: Observe, learn, understand - Then: Do! - Concepts and principles not tailored to Shiny apps and specific tools / services ] --- ## Workshop prerequisites - A GitHub account is necessary for the workshop: also make sure you have its password at hand - Git must be installed on your computer - A recent version of RStudio (>= 1.2) is recommended - R installation: R >= 3.6.x, ideally R 4.0.x - Packages pre-installation: - https://github.com/miraisolutions/eRum2020Workshop-prereqs#readme - Optional Git-related tools - [compareWith](https://github.com/miraisolutions/compareWith#readme) R package - [Sublime Merge](https://www.sublimemerge.com) --- ## (Shiny) App development cycle Target "Production": deployed and made accessible .pull-right[ <img src="./img/devCycle.001.png" width="35%" style="position:absolute;top:60px;right:50px;" /> ] - Development (app code) - ... - Deployed app (today: shinyapps.io) Fill the dots: - Safety checks and controls - Collaboration and planning - Automate! .center[ ### Head start: Morning workshop<br/>[_Is R ready for Production?<br/>Let's develop a Professional Shiny Application!_](https://github.com/Milano-R/erum2020program#is-r-ready-for-production-lets-develop-a-professional-shiny-application) ] --- ## Head start: Packaged Shiny app on GitHub Shiny app developed as **R package** under **version control** - Best place for R functions to live in => modularization - Documentation, namespaces / dependencies, tests, sanity checks, controls, ... - Package [**`golem`**](https://thinkr-open.github.io/golem) providing development framework<br/> `golem` : Shiny app = `devtools` (`usethis`) : R package - Package dependencies explicitly tracked using [**`renv`**](https://rstudio.github.io/renv) - Run the app: install the package and call a function! - **GitHub** repo to share the source code and track changes **Production readiness** - How to ensure it is maintained, regularly checked and ensured? - How to control what is deployed to "production" (the live app)? ** => Automated, streamlined pipelines** - Security, Quality, Stability - Optimized for speed of delivery --- ## Version control - Git(Hub) .pull-left[ - Git is a popular, free and open source version control system designed to be very efficient and to support distributed, non-linear workflows - With GitHub remote hosting, it is the _de facto_ standard for open source (R) projects - Track changes - History - Share and Collaborate ] .pull-right[ - Pull ([`git pull`](https://git-scm.com/docs/git-pull)): Incorporate changes from a remote repository into the current local branch - Stage ([`git add`](https://git-scm.com/docs/git-commit)) changes: Prepare the content for the next commit - Commit ([`git commit`](https://git-scm.com/docs/git-commit)): Record staged changes to the current branch - Push ([`git push`](https://git-scm.com/docs/git-push)): Update remote repository using the local branch ] Resources - [Git documentation](https://git-scm.com/doc) - [Happy Git and GitHub for the useR](https://happygitwithr.com/), Jenny Bryan, Jim Hester - [Git and GitHub](http://r-pkgs.had.co.nz/git.html), R packages, Hadley Wickham --- ## Running example - (The app developed during the morning workshop) - A somewhat **minimal example** of a Shiny app showcasing the same production-readiness approach - GitHub repo https://github.com/miraisolutions/ShinyCICD-min - Create your own copy of the repo by [forking](https://github.com/miraisolutions/ShinyCICD-min/fork) it to your GitHub user account **_Clone_** locally your forked repo (https://github.com/&lt;user&gt;/ShinyCICD-min) as a **new RStudio project** ``` File > New Project... > Version Control > Git ``` --- ## `renv` dependencies The project is based on [`renv`](https://rstudio.github.io/renv) - Project-specific dependencies are tracked via an `renv.lock` file (+ `.Rprofile` and `renv/`) - Ensure the set of dependencies is collaboratively shared an maintained (and can be used to align production deployments) - When you open the project (and a new R session starts), `renv` detects the project dependencies are out of sync ```r renv::status() renv::restore() ``` -- **Note**: If you are on R 3.6.x, you will see a warning `Project requested R version '4.0.z' but '3.6.x' is currently being used`. No big deal, simply align the R version in the lockfile via ```r renv::snapshot() ``` What changes (check the Git pane)? Should you commit the modifications? --- ## Development initialization - `install.packages("devtools")` - Adapt the README to your forked repo: `<user>/ShinyCICD-min` - Diff: RStudio, compareWith, Sublime Merge ```r renv::install("miraisolutions/compareWith") # remotes::install_github("miraisolutions/compareWith") w/o renv ``` Stage > Commit > Push --- ## Install the package, run the app! RStudio's Build Pane - Install and Restart (Ctrl+Shift+B) Then, launching the app is just about calling a function from the package: ```r shinyCICD::run_app() ``` Easy, isn't it? - Dependencies under control (esp. with distributed users) - No need to source the app code (once the package is installed, the source package code does not even matter!) Speed up in a development workflow - `devtools::load_all()` (Ctrl+Shift+L): mimics the full-install, but is much quicker - `golem::document_and_reload()` includes a few extras (`help()`) --- ## R's package check suite - `R CMD check` built into R, conveniently accessed via - ```{r} devtools::check() ``` - ```{r} rcmdcheck::rcmdcheck() ``` - RStudio: `Build Pane` > `Check` Everything should be fine Familiarize with the number of checks (see http://r-pkgs.had.co.nz/check.html for reference) - If the check get stuck, it might be due to issues with `renv` (mainly on Windows) - In case, fall-back on `renv`-free setup and still follow the rest of the workshop - `renv::deactivate()` - delete `renv.lock` and the `renv` folder --- ## Automated checks on Travis CI [ <img src="./img/TravisCI.png" width="50px" style="vertical-align: text-bottom;" /> Travis CI ](https://travis-ci.com/) - Open-source hosted continuous integration service - Automate package-level checks for continuous integration, triggered upon any push event (and pull requests) on the GitHub repo - Login at https://travis-ci.com/ (using your GitHub account) - Activate authorization via GitHub Apps integration .center[ <img src="./img/TravisActivate.png" width="450px" style="position: relative;" /> <img src="./img/TravisInstall.png" width="250px" style="position: relative;" /> <img src="./img/TravisSetup.png" width="250px" style="position: relative;" /> ] --- ## Automated checks on Travis CI - Setup Travis for your project ```r usethis::use_travis() # ext = "com" for usethis < 1.6.0 ``` - This will bring you to https://travis-ci.com/account/repositories, from where you can "Manage repositories on GitHub" to make sure Travis CI has access to your repo - Look at the R console and the Git diffs (**don't commit changes yet**) - which files were changed by `usethis::use_travis()`? -- Travis has a default pipeline for R (https://docs.travis-ci.com/user/languages/r) Align R _major_._minor_ version to the project setup (spaces matter!) ```yaml r: - 4.0 # or 3.6 ``` Use `renv` to restore dependencies: - `cache` and `install` field as in the [Using renv with Continuous Integration](https://rstudio.github.io/renv/articles/ci.html#travis-ci) vignette --- ## Automated checks on Travis CI ```yml language: R r: - 4.0 # or 3.6 cache: directories: - $HOME/.local/share/renv - $TRAVIS_BUILD_DIR/renv/library install: - Rscript -e "if (!requireNamespace('renv', quietly = TRUE)) install.packages('renv')" - Rscript -e "renv::restore()" ``` **Commit & push** all changes - This will trigger the first run on Travis, which can take >~10 minutes - Caching will speed up future builds - Go to the GitHub repo (`usethis::browse_github()`) - Do you find any indication of the CI status for the commit? - Scroll through the Travis log --- ## GitFlow .pull-left[ [GitFlow](https://nvie.com/posts/a-successful-git-branching-model/) is a popular and successful branching model that enable collaborative, controlled development and releases, assuring the safety and quality of the released product According to the GitFlow approach, the central repo holds two main branches with an infinite lifetime: - `master`: reflects a production-ready state - `develop`: reflects a state with the latest delivered developments for the next release In addition - `feature/`, `release/`, `hotfix/` branches ^{[https://nvie.com/posts/a-successful-git-branching-model]} ] <img src="./img/gitflow.png" width="35%" style="position: absolute; right: 100px; top: 60px;" /> --- ## Pull requests and branch control It is important that the code is checked before it is merged back into the main development / release line, to ensure robustness and stability of the code. This can be ensured by [setting restrictions on branches](https://help.github.com/en/github/administering-a-repository/enabling-branch-restrictions): .pull-left[ - Settings > Branches - `develop` becomes the default branch - Branch protection for `master` and `develop` - Commits are pushed only via [pull requests](https://help.github.com/en/github/collaborating-with-issues-and-pull-requests/about-pull-requests) - [Require status checks](https://help.github.com/en/github/administering-a-repository/about-required-status-checks) (e.g. Travis CI, GitHub Actions) to pass before merging a pull request - [Require pull request reviews](https://help.github.com/en/github/collaborating-with-issues-and-pull-requests/requesting-a-pull-request-review) from collaborators before merging ] .pull-right[ <img src="./img/branches-control.001.png" width="100%" /> ] --- ## Issues, projects, pull requests Track development items as GitHub [Issues](https://help.github.com/en/github/managing-your-work-on-github/creating-an-issue) and [Projects](https://help.github.com/en/github/managing-your-work-on-github/about-project-boards) (visual boards) - **NOTE** for forked repos, Issues / Projects are typically disabled by default => Repository **Settings** GitFlow - `feature/` branches for developing individual issues in isolation - Development reviewed and safely integrated into `develop` via [pull request](https://help.github.com/en/github/collaborating-with-issues-and-pull-requests/about-pull-requests) .center[ <img src="./img/project-board.png" width="80%" /> ] --- ## Agile development GitFlow fits well with Agile frameworks - During a development cycle a set of Issues is worked on according to prioritization - At the end of each development cycle it is possible to perform a release and have a development increment onto production - Each feature can be inspected and feedback can be given before its release, keeping the final product always up to date with the end user needs .center[ <img src="./img/agile_dev.001.png" width="80%" /> ] --- ## Deployment: shinyapps.io [shinyapps.io](https://www.shinyapps.io) hosting service offered by RStudio Login at https://www.shinyapps.io/admin using your GitHub account [Configure rsconnect](https://docs.rstudio.com/shinyapps.io/getting-started.html#configure-rsconnect) and deploy your app from your local R installation ```r golem::add_shinyappsio_file() install.packages(c("rsconnect", "pkgload")) ``` ```r rsconnect::setAccountInfo(...) ``` ```r # we explicitly exclude the renv directory (as well as hidden files) rsconnect::deployApp(appName = "ShinyCICD", appFiles = setdiff(list.files(), "renv")) ``` ```r usethis::use_git_ignore("rsconnect") usethis::use_package("rsconnect", "Suggests") renv::snapshot() ``` --- ## GitHub Actions and Continuous Deployment See the [dedicated chapter in our Technical Guidelines](https://mirai-solutions.ch/techguides/github-actions.html) --- ## Release A [Release](https://help.github.com/en/github/administering-a-repository/releasing-projects-on-github) is the deployment to production of a consolidated set of development. Each new release determines a new version (minor or major) of the software. According to the GitFlow approach, no work should be pushed directly to `master`, which is updated as part of a release with a pull request from a stable `develop` branch or a `release` branch. <img src="./img/release.001.png" width="20%" style="position:absolute;bottom:100px;right:70px;" /> --- ## How to do a release We want to release the new feature(s) as a [major/minor/patch release](http://r-pkgs.had.co.nz/release.html#release-version) - Create release branch `release/vX.Y.Z` (e.g. `v1.2.1` for a _minor_ release) from `develop`, where the package version includes a fourth development component `9000` (e.g. `1.1.1.9000`) - Update the package version via `usethis::use_version()` locally on branch `release/v1.2.1` - Consolidate the changelog (`NEWS.md` in R, see e.g. [recommendations](http://r-pkgs.had.co.nz/release.html#important-files) and [style guide](https://style.tidyverse.org/news.html#news-release)) - Commit & push - Create a pull request to `master` on GitHub, get it approved and merged - Create release on GitHub as "v1.2.1", with title "ShinyCICD 1.2.1" and content of the corresponding `NEWS.md` section - Go back to a development version `1.2.1.9000` `usethis::use_dev_version()`, still on branch `release/v1.2.1` - Commit & push - Create a pull request to `develop` on GitHub, get it approved and merged - Delete the release branch --- ## Other Mirai's resources - Docker-based Shiny app cloud deployment + integration into a custom website gallery - [slides](http://www.user2019.fr/static/pres/t258350.pdf) and [recording](https://youtu.be/DizjPeNTeeo) @ ToulouseR!2019 - Packaged bookdown website with Travis-based CI/CD - https://github.com/miraisolutions/vbzdelays - Part of a [_MiraiLabs_ workshop](https://github.com/miraisolutions/MiraiLabs/tree/master/from-prototype-to-production#readme) - e-Rum2020 [CovidR contest](https://milano-r.github.io/erum2020-covidr-contest/) - Open source repository collecting contributions via PR - Inclusion of contributions in an [rmdgallery](https://riccardoporreca.github.io/rmdgallery/) website - Automation: website update upon PR, handling of voting issues, awards and badges, maintenance, ... --- ## Get in touch **Other Services by Mirai solutions:** - Training customized to your business case - technical and agile topics - Data Analytics & Software development: from your needs to a functional implementation - Infrastructure design, management and maintenance **Get in touch:** - LinkedIn https://www.linkedin.com/company/mirai-solutions-gmbh/ - email info@mirai-solutions.com - website https://mirai-solutions.ch <img src="./img/mirai-solutions-logo-flyer.png" width="40%" style="position:absolute;bottom:80px;right:50px;" /> --- class: inverse center middle # Thank you!