Roadmap v6.0.0 release notes

Roadmap v6.0.0 upgrades the application to Ruby 3.4 and Rails 8.1, with compatible dependency updates and changes to support the new framework. This release introduces no new product features or API versions.

Platform updates

Application compatibility

• Adapted authentication, enum definitions, routing, exports and database schema handling for Rails 8.1.
• Updated authentication dependencies, including Devise, Devise Invitable and OmniAuth CSRF protection.
• Updated supporting libraries for pagination, file storage, email, database connections and application serving.
• Preserved support for PostgreSQL and MySQL.

Development and testing

• Updated CI to Node.js 24 and current GitHub Actions runtimes.
• Pinned Chrome in CI to improve browser test reliability.
• Added schema drift checks and regression coverage.
• Updated RSpec, Selenium, linting and security analysis dependencies.
• Replaced annotate with annotaterb.
• Removed Spring and its associated integrations.

Upgrade requirements

Update deployment and development environments to the runtime versions listed above. Install the updated Ruby and JavaScript dependencies, run database migrations, and rebuild application assets as part of the upgrade.

Development workflows that invoke Spring should use the Rails and RSpec commands directly.

• Patch API token auth to check user's status #3606
• fix(policy): harden authorization for plan resources #3613

• Bump Ruby to v3.1.4 and use .ruby-version in CI
  • #3566
  • #3573
• Enable session timeout after 90 minutes of inactivity #3568
• Validate CSV Separator #3569
• Fix rendering of confirm_merge partial #3567
• Improve CI configuration #3551
• Bump nokogiri from 1.17.2 to 1.18.9 #3547
• Bump activerecord from 7.1.5.1 to 7.1.5.2 #3549
• Bump rack-session from 2.1.0 to 2.1.1 #3524
• Bump net-imap from 0.4.19 to 0.4.20 #3518
• Bump rexml from 3.4.1 to 3.4.2 #3562
• Bump tmp from 0.2.3 to 0.2.4 #3548
• Update Gems and Address New Rubocop Offences #3572

v5.0.1

• Updated seeds.rb file for identifier_schemes to include context value and removed logo_url and idenitifier_prefix for Shibboleth (as it was causing issues with SSO). #3525
• Adjustments to style of select tags and plan download layout #3509
• Fix failing eslint workflow / upgrade actions/checkout & actions/setup-node to v3 #3503
• Remove Auto-Generated TinyMCE Skins and Add public/tinymce/skins/ to .gitignore #3466

The primary change in this major DMPRoadmap release is the upgrade from Rails 6.1 to Rails 7.1. As part of this upgrade, the handling of conditional questions was updated, including a migration that converts the Condition.option_list and Condition.remove_data fields to proper JSON types (#3497).

In addition to the framework upgrade, this release includes a number of refactors, bug fixes, dependency updates, and performance optimisations. For further details, please refer to the pull requests and issues listed below:

• Updated app to Rails 7 #3426, #3496
• Address Some Bullet Warnings / Optimise Mean Request Times #3440
• Fix Flaky Tests / Optimize Checking of plan.title Within spec/features/plans/exports_spec.rb #3451
• Refactor Plan.deep_copy(plan) #3469
• Fixed a bug in the deep copy of plans where the old identifier was being copied into the new plan. We now copy the generated id of the new plan to the identifier field.
• Fixed bar chart click function in the Usage dashboard (GitHub issue #3443)
• Fixed broken link for the V1 API documentation.
• Fix hidden_field_tag Nested Attributes Format For Rails 7 Upgrade and Add Test Coverage #3479
• Update all workflows to runs-on: ubuntu-24.04 and Consolidate Capybara Config #3487
• Add pdf handling in render_respond_to_format_with_error_message #3482
• Lower PostgreSQL GitHub Action Chrome Version to Address Breaking Changes Between Latest Chrome Version (134) and /features Tests #3491
• Bumped dependencies via bundle update && yarn upgrade #3483
• Fixed issues with Conditional Question serialization offered by @briri from PR CDLUC3#667 for DMPTool. There is a migration file with code for MySQL and Postgres to update the Conditions table to convert JSON Arrays in string format records in the conditions table so that they are JSON Arrays.
• Refactor org_admin/conditions/_form.html.erb #3502
• Refactor Question.save_condition #3501
• Bump tinymce from 6.8.5 to 7.2.0 #3505

Note this upgrade is mainly a migration from Bootstrap 3 to Bootstrap 5.

Note that this will have a significant impact on any scss and html customizations you may have made to your fork of this project.

The following links will be helpful:

Get started with Bootstrap v5.2.3

Migrating to v4

How to Migrate from Bootstrap Version 3 to 4

Migrating to v5

How to Migrate from Bootstrap Version 4 to 5

Use Bootstrap 5 with Ruby on Rails 6 and webpack

What happened to $grid-float-breakpoint in Bootstrap 4. And screen size breakpoint shift from 3 -> 4

What are media queries in Bootstrap 4?

Key changes

• Node package changes:
  • Changed version of bootstrap "^3.4.1" --> "^5.2.3"
  • Added @popperjs/core.
  • Removed bootstrap-3-typeahead, bootstrap-sass & popper.js
• Stylesheet changes

  • In app/assets/stylesheets/application.scss:

    • removed bootstrap-sass import
      
      and replaced with
      
      @import "../../../node_modules/bootstrap/scss/bootstrap";

  • The order of the import statements have been changed to import the blocks/ and utils/ after the default bootstrap stylesheets

  • In app/assets/stylesheets/blocks/:

    • Replaced in relevant files:
      • @use "../../../../node_modules/bootstrap-sass/assets/stylesheets/_bootstrap.scss" as * ;
        
        with
        
        @use "../../../../node_modules/bootstrap/scss/bootstrap" as *;
    • Enclosed all division calculations using symbol / with calc() function,
      
      e.g., replaced
      
      padding-right: $grid-gutter-width / 2;
      
      with
      
      padding-right: calc($grid-gutter-width / 2);
      
    • Replaced breaking media queries since Bootstrap 3:

      • @media (max-width: $grid-float-breakpoint-max) {}
        
        with
        
        @include media-breakpoint-down(md){}

      • @media (max-width: $grid-float-breakpoint-max) {}
        
        with
        
        @include media-breakpoint-down(md) {}

  • Deleted app/javascript/src/utils/popoverHelper.js.

• Mixins
  • Media query mixins parameters have changed for a more logical approach.
    • media-breakpoint-down() uses the breakpoint itself instead of the next breakpoint (e.g., media-breakpoint-down(lg) instead of media-breakpoint-down(md) targets viewports smaller than lg).
• Color system
  • All lighten()and darken() functions replaced. These functions will mix the color with either white or black instead of changing its lightness by a fixed amount.
    • Replaced lighten() by tint-color().
    • Replaced darken() by shade-color().

Components & HTML

Note many of these Bootstrap changes has required us to rewrite or change some of the Javascript files.

When we use a native DOM element in Javascript, we obtain it by applying get() to the Jquery element (cf., https://api.jquery.com/get/).
We sometimes use the button native Dom element to programmatically click, as the Jquery button element with trigger('click') won't work because the trigger() function cannot be used to mimic native browser events, such as clicking (cf., https://learn.jquery.com/events/triggering-event-handlers/ )

• Accordion & spinners
  • Bespoke versions replaced by Bootstrap 5 accordion and spinner now.
  • Accordion
    • Changed the default Bootstrap arrow icon for the accordion to use the fontawesome icons plus and minus icons. Created a several accordion specific colour variables:
      
      // Accordion colors
      
      $color-accordion-button: $color-primary-text;
      
      $color-accordion-button-icon: $color-primary-text;
      
      $color-accordion-button-bg: $color-primary-background;
      
      $color-accordion-button-active-bg: shade-color($color-accordion-button-bg, 30%);
      
      (See app/assets/stylesheets/blocks/_accordion.scss and app/assets/stylesheets/variables/_colours.scss for details.)
    • The drag icon in app/views/org_admin/sections/_section.html.erb now appears after the plus (or minus) icon.
  • The spinner block now uses classd-none instead ofhidden to hide.
  • In views with multiple accordion sections with "expand all" or "collapse all" links, we use the native Dom element of the accordion buttons to programmatically click, (cf. to note above).
• Buttons
  • Bootstrap dropped btn-block class for utilities. So we removed any styling using it.
  • Close Buttons: Renamed close tobtn-close.
  • Renamed btn-default to btn-secondary and variable $btn-default-color changed to $btn-secondary-color.
• Dropdowns
  • Dropdown list items with class dropdown have class dropdown-item added usually withpx-3 for positioning.
  • Added new dropdown-menu-dark variant and associated variables for on-demand dark dropdowns.
  • Data attributes changes required by Bootstrap 5 (as used by accordion and dropdown buttons):
    • data-display --> data-bs-display
    • data-parent --> data-bs-parent
    • data-target --> data-bs-target
    • data-toggle --> data-bs-toggle
  • Bootstrap 5 Popover added to some dropdown-menu items by adding attribute data-bs-toggle="popover"
• Form
  • form-group class replaced with form-control.
  • Form labels now require form-label or form-check-label to go with form-control and form-check respectively. So all obsolete control-label replaced by form-label and missing ones added.
  • Dropped form-specific layout classes for our grid system. Use Bootstrap grid and utilities instead of form-group, form-row, or form-inline.
  • form-text no longer sets display, allowing you to create inline or block help text as you wish just by changing the HTML element.
  • Input group addons are now specific to their placement relative to an input. So input-group-addon and in our case we replaced with
    input-group-addon.
  • Renamed checkbox and radio into form-check.
• Images
  • Renamed img-responsive to img-fluid.
• Labels and badges
  • Class label has been removed and replaced by badge to disambiguate from the <label> element.
    • Renamed label class to badge
    • Replaced label-default by bg-secondary
    • Replaced label-info by bg-info
    • Replaced label-warning by bg-warning .text-dark
    • Replaced label-danger by bg-danger
• Links
  • Links are underlined by default (not just on hover), unless they're part of specific components. So we had to add css to remove underline in many cases.
• Modals
  • To programmatically show or hide a Bootstrap modal, we have followed both these approaches:
    • Either, get access to the Jquery modal element and call functions modal('show') or modal('hide').
    • Or, apply click() to the native Dom element of the button to trigger the modal (cf. to note above).
• Navs & navbars
  • Bootstrap rewrote component with flexbox. Dropped nearly all > selectors for simpler styling via un-nested classes.
    Instead of HTML-specific selectors like .nav > li > a, we use separate classes for navs, nav-items, and nav-links. (Note because the nav-link class has not always been added as it comes with styles not appropriate for our styling for links.)
    This makes your HTML more flexible while bringing along increased extensibility. So we have dropped HTML-specific selectors and css in _navs.scss
    e.g.,
    
    .nav-tabs > li > a:hover --> nav-tabs nav-link:hover,
    
    .nav-pills > li > a:hover -->nav-pills .nav-link:hover.
    • Pages with css classes nav andnavbar updated to work with Bootstrap 5. So app/assets/stylesheets/blocks/_navbars.scss and app/assets/stylesheets/blocks/_navs.scss updated.
      • Replacednav navbar-nav combination --> navbar-nav
      • Replacednavbar-toggle --> navbar-toggler
      • Replaced multiple spans innavbar-toggle button with classicon-bar
        --> single span withtoggler-icon
      • Lists with nav navbar-nav have classnav-item added to list elements.
    • Note because the nav-link class include styling that is not appropriate in many places, we have not included it in those cases.
• Notifications
  • Notifications now use classes d-block and d-none to show and hide respectively.
• Panels, thumbnails & wells (replacements)
  • Bootstrap 5 dropped panels, thumbnails and wells. So pages with them updated with Bootstrap 5 replacements.
    • All views with css classespanel, panel-body, panel-* Have panel replaced by card to give card, card_body, card-*, etc.
    • As panel-default and some otherpanel css classes don't have card equivalents with same suffixes we have added these classes temporarily in _cards.sccs, e.g.,.card-default, etc.
• Utilities
  • Bootstrap renamed several utilities to use logical property names instead of directional names with the addition of RTL support:
    • Renamed left-* and right-* to start-* and end-*.
    • Renamed float-left and float-right to float-start and float-end.
    • Renamed ml-* and mr-* to ms-* and me-*.
    • Renamed pl-* and pr-* to ps-* and pe-*.
    • Renamed text-left and text-right to text-start and text-end.
  • The hidden and show classes have been removed because th...

Added

• Added MORE_INFO and LESS_INFO JS constants (for the Research Outputs feature)
• Added a .gitkeep file to the app/assets/builds directory to address potential issues when building the application.css file during production deploys - #3314

Fixed

• Updated the default font on the 'Download page' to be 'Roboto, Arial, Sans-Serif' - #3337
• Fixed an issue with API V0 that was causing a 500 Internal Server error - #3316
• Solved issue where spring was loaded in production mode : ran bin/spring binstub --all
• Updated fontawesome to V6 - #3328

Note this upgrade is a migration from Ruby v2.7.6 to v3.0.5. Note that this could have an impact on any customizations you may have made to your fork of this project. Please see https://www.fastruby.io/blog/ruby/upgrades/upgrade-ruby-from-2.7-to-3.0.html for further information on what to check. In particular, please note the changes to the way Ruby 3 handles keyword arguments

Note that the Webpacker gem has been removed in favor of jsbundling-rails. This was done in preparation for the future migration to Rails 7. See issue #3185 for more details on this change. If, after migrating to this version, you see 'Sprockets' related errors in your application you will need to rebuild you asset library. To do this run bin/rails assets:clobber && bin/rails assets:precompile from the project directory.

All gem and JS dependencies were also updated via bundle update && yarn upgrade

Enhancements

• Added enum to the funding status attribute of plan model to make the dropdown of 'funding status' being translatable
• Allow users to download both single phase and in PDF, TEXT and DOCX format. CSV file can only download single phase instead of all phases.

Developers

When running the application on your development machine, run bin/dev instead of the old rails s approach. This will kick off 'foreman' which will startup jsbuilding and cssbuilding so that they watch your CSS and JS files for changes.

Pay close attention to your customizations in the app/views/branded/ directory and any custom JS or CSS you may have for your instance. You may need to make modifications to work with the new JS/CSS Bundling methods. Most notably:

• The application.js and application.css files have moved. If you were including your custom JS or CSS in this file, make sure that the new versions are importing your files.
• If you have custom SCSS files in a separate directory (e.g. not in blocks, variables, etc.), You will need to add an _index.scss file for your directory. See the one in /app/assets/stylesheets/blocks/ for an example.
• If you reference any variables (e.g. $color-primary-background) in your custom SCSS files, you will need to add @use '../variables/colours' as *; at the top of the file.
• The way we include the JS and CSS into the ERB has changed. If you have a custom app/views/branded/layouts/application.html.erb make sure that you update the javascript and stylesheet link tags in your <head> section. See app/views/layouts/application.html.erb.
• Note that the app/javascript/src/utils/tinymce.js file extension has changed from js.erb to js. If you reference this file in your custom JS, you will need to update your import statement.

Upgrade to Ruby 3

• Upgrade to Ruby version 3.0.5 #3225
• Bumped all Github actions to use ruby 3.0
• Removed .freeze from Regex and Range constants since those types are already immutable
• Fixed Rubocop complaint about redundancy of r.nil? ? nil : r.user, so changed it to r&.user in app/models/plan.rb
• Fixed Rubocop complaint about redundant :: in config.log_formatter = ::Logger::Formatter.new in config/environments/production.rb
• Froze lib/deprecators/*.rb constants that were Strings
• Updated places that were incorrectly using keyword args. See this article for an overview

Upgraded TinyMCE to v6

• Upgraded TinyMCE to v6 (v5 EOL is April 20 2023)
• Adjusted JS code to conform to new TinyMCE version
• Adjusted views to work with the new version
• Updated variables.scss file to fix issue with button text/background color contrast
• Updated blocks/_tables.scss to fix issue with dropdown menu overlap against table
• updated config/initializers/assets.rb to copy over the tinymce skins and bootstrap glyphicons to the public directory so that they are accessible by TinyMCE and Bootstrap code

Removed webpacker gem

As Webpacker is no longer maintained by the Rails community, we have replaced it by jsbundling-rails and cssbundling-rails for the Javascript & CSS compilation.

• Removed webpacker gem
• Added jsbundling-rails
• Updated webpack and its configuration to V5
• Moved app/javascript/packs/application.js to app/javascript/application.js
• Removed ruby-sass gem
• Added cssbundling-rails gem and DartSass JS library
• Updated SASS stylesheets following the migration to the latest version of the sass package (See below).
• Removed font-awesome-sass gem and used @fortawesome/fontawesome-free npm package
• Issue with @import 'font-awesome-sprockets'; line in app/assets/stylesheets/application.scss. Removed that line after referring to the latest font-awesome install/setup guide which no longer includes it.

With the removal of Webpacker, the Javascript/SASS code is no longer automaticaly compiled when using the rails server command. It has been replaced by the bin/dev command that launch the rails server and the processes that watch for changes in the SASS and Javascript code.

SASS update : removal of the @import keyword

With the removal of the webpacker gem, the DartSass package has been installed to ensure the compilation of the Sass stylesheet and with it, an update to the Sass version used by the code :

• @import keyword for custom stylesheets has been removed (although we can still import stylesheets from externals packages) and has been replaced by @use and @forward
• An _index.scss file have to be created in folders containing multiple sass files. Each file have to be included in the index with the @use or @forward keyword.
• In most cases @import can be replaced by @use when importing a file.
• @forward makes mixins, functions and variables available when a stylesheet is loaded.
• When imported, Sass variables are now namespaced with the file name in which they are declared (ex : color: colors.$red). A namespace can be renamed (ex : @use "colours" as c;) or removed when included (ex : @use "colours" as *;)
• Sass variables are no longer declared globally and have to be included in files where they are used.
  For more detailed explanation, please refer to this video : https://www.youtube.com/watch?v=CR-a8upNjJ0

Introduction of RackAttack

Rack Attack is middleware that can be used to help protect the application from malicious activity. You can establish white/black lists for specific IP addresses and also define rate limits.

• Using Rack-attack address vulnerabilities pointed out in password reset and login: there was no request rate limit.#3214

Cleanup of Capybara configuration

• Cleaned up Gemfile by:
  • removing gems that were already commented out
  • removed selenium-webdriver and capybara-webmock
  • removing version restrictions on: danger, font-awesome-sass, webdrivers
• Cleaned up spec/rails_helper.rb and spec/spec_helper.rb
• Simplified the spec/support/capybara.rb helper to work with the latest version of Capybara and use its built in headless Chrome driver

Rubocop updates

• Installed rubocop-performance gem and made suggested changes
• Added lib tasks as exclusive from debugger rubocop check after rubocop upgrading to >= v1.45 #3291

GitHub actions updates

• Added node version specification (v16) to eslint, PostgreSQL and MySQL github action to eliminate digital routine enveloped error #319

New Rake task for updating dependencies

• Added a new bin/rails update:all task that will update the gem and JS dependencies and also add the x86_64-linux environment to the bundle's supported platform list

Bug Fixes

• Updated sans-serif font used in PDF downloads to Roboto since Google API no longer offers Helvetica
• Fixed discrepencies with default/max per_page values for API and UI pagination
• Updated JS that used to call the TinyMCE setMode() function so that it now calls mode.set() because the former is now deprecated.
• Patched an issue that was causing a template's visibility to change to 'organizationally_visible' when saving on the template details page.
• Fixed an issue with the Rails 6 keyword arguments change that was causing the paginable_sort_link to fail

Changes

• Added validation with custom error message in research_output.rb to ensure a user does not enter a very large value as 'Anticipated file size'. #3161
• Added popover for org profile page and added explanation for public plan
• Updated the CSV export so that it now includes research outputs
• Change csv file name for statistics from 'Completed' to 'Created'

Developers

• Added CHANGELOG.md and Danger Github Action #3257
• Froze mail gem version #3254
• Added scss files to EditorConfig
• Updated Ruby & JS dependencies

Bug Fixes

• Fixed bin/rails db:schema:load issue in version 4.0.0 and enhanced bin/setup command to improve the installation process #3216
• Added patches to 4.0.0 version for Rails 6 fixes
  • Removed translation.io error message from the system email to users
  • Added new translations to usage page, feedback notification page and guidance group page
• Accessibility fixes #3211
  • Missing form labels - Contact us
  • Missing form label - Dashboard - test checkbox
  • Missing form labels - Create plan
  • Missing form label
  • Also went through and fixed errors on Project details, Contributor form, Research Output form and Write Plan
• Fixed potential plan visibility issue
• Added header translation and removed the extra 'Affiliation' row in the downloaded csv file
• Fixed Tried to load unspecified class: Symbol error that some users encountered in download page

Developers

• Added EditorConfig file to help maintain constant coding style
• Updated some coding style #3249
• Kept Github Actions using ubuntu-20.04 as the OS for test #3247

Dependency Updates

• Bumped engine.io from 6.2.0 to 6.2.1
• Bumped Webdriver from version 4 to 5.2
• Bumped nokogiri from 1.13.18 to 1.13.9