An original co-founder of Agaric, Dan spends his time and energy building on this mystical phenomenon popularly called the Internet. He believes in the principles of free open source software and develops primarily using the Drupal content management framework.
Find his latest at dhakimzadeh.com.
Today we will learn how to migrate dates into Drupal. Depending on your field type and configuration, there are various possible combinations. You can store a single date or a date range. You can store only the date component or also include the time. You might have timezones to take into account. Importing the node creation date requires a slightly different configuration. In addition to the examples, a list of things to consider when migrating dates is also presented.
Getting the code
You can get the full code example at https://github.com/dinarcon/ud_migrations The module to enable is UD date whose machine name is ud_migrations_date. The migration to execute is udm_date. Notice that this migration writes to a content type called UD Date and to three fields: field_ud_date, field_ud_date_range, and field_ud_datetime. This content type and fields will be created when the module is installed. They will also be removed when the module is uninstalled. The module itself depends on the following modules provided by Drupal core: datetime, datetime_range, and migrate.
Note: Configuration placed in a module’s config/install directory will be copied to Drupal’s active configuration. And if those files have a dependencies/enforced/module key, the configuration will be removed when the listed modules are uninstalled. That is how the content type and fields are automatically created.
PHP date format characters
To migrate dates, you need to be familiar with the format characters of the date PHP function. Basically, you need to find a pattern that matches the date format you need to migrate to and from. For example, January 1, 2019 is described by the F j, Y pattern.
As mentioned in the previous post, you need to pay close attention to how you create the pattern. Upper and lowercase letters represent different things like Y and y for the year with four-digits versus two-digits, respectively. Some date components have subtle variations like d and j for the day with or without leading zeros respectively. Also, take into account white spaces and date component separators. If you need to include a literal letter like T it has to be escaped with \T. If the pattern is wrong, an error will be raised, and the migration will fail.
Date format conversions
For date conversions, you use the format_date plugin. You specify a from_format based on your source and a to_format based on what Drupal expects. In both cases, you will use the PHP date function's format characters to assemble the required patterns. Optionally, you can define the from_timezone and to_timezone configurations if conversions are needed. Just like any other migration, you need to understand your source format. The following code snippet shows the source and destination sections:
source:
plugin: embedded_data
data_rows:
- unique_id: 1
node_title: 'Date example 1'
node_creation_date: 'January 1, 2019 19:15:30'
src_date: '2019/12/1'
src_date_end: '2019/12/31'
src_datetime: '2019/12/24 19:15:30'
destination:
plugin: 'entity:node'
default_bundle: ud_dateNode creation time migration
The node creation time is migrated using the created entity property. The source column that contains the data is node_creation_date. An example value is January 1, 2019 19:15:30. Drupal expects a UNIX timestamp like 1546370130. The following snippet shows how to do the transformation:
created:
plugin: format_date
source: node_creation_date
from_format: 'F j, Y H:i:s'
to_format: 'U'
from_timezone: 'UTC'
to_timezone: 'UTC'Following the documentation, F j, Y H:i:s is the from_format and U is the to_format. In the example, it is assumed that the source is provided in UTC. UNIX timestamps are expressed in UTC as well. Therefore, the from_timezone and to_timezone are both set to that value. Even though they are the same, it is important to specify both configurations keys. Otherwise, the from timezone might be picked from your server’s configuration. Refer to the article on user migrations for more details on how to migrate when UNIX timestamps are expected.
Date only migration
The Date module provided by core offers two storage options. You can store the date only, or you can choose to store the date and time. First, let’s consider a date only field. The source column that contains the data is src_date. An example value is '2019/12/1'. Drupal expects date only fields to store data in Y-m-d format like '2019-12-01'. No timezones are involved in migrating this field. The following snippet shows how to do the transformation.
field_ud_date/value:
plugin: format_date
source: src_date
from_format: 'Y/m/j'
to_format: 'Y-m-d'Date range migration
The Date Range module provided by Drupal core allows you to have a start and an end date in a single field. The src_date and src_date_end source columns contain the start and end date, respectively. This migration is very similar to date only fields. The difference is that you need to import an extra subfield to store the end date. The following snippet shows how to do the transformation:
field_ud_date_range/value: '@field_ud_date/value'
field_ud_date_range/end_value:
plugin: format_date
source: src_date_end
from_format: 'Y/m/j'
to_format: 'Y-m-d'The value subfield stores the start date. The source column used in the example is the same used for the field_ud_date field. Drupal uses the same format internally for date only and date range fields. Considering these two things, it is possible to reuse the field_ud_date mapping to set the start date of the field_ud_date_range field. To do it, you type the name of the previously mapped field in quotes (') and precede it with an at sign (@). Details on this syntax can be found in the blog post about the migrate process pipeline. One important detail is that when field_ud_date was mapped, the value subfield was specified: field_ud_date/value. Because of this, when reusing that mapping, you must also specify the subfield: '@field_ud_date/value'. The end_value subfield stores the end date. The mapping is similar to field_ud_date except that the source column is src_date_end.
Note: The Date Range module does not come enabled by default. To be able to use it in the example, it is set as a dependency of demo migration module.
Datetime migration
A date and time field stores its value in Y-m-d\TH:i:s format. Note it does not include a timezone. Instead, UTC is assumed by default. In the example, the source column that contains the data is src_datetime. An example value is 2019/12/24 19:15:30. Let’s assume that all dates are provided with a timezone value of America/Managua. The following snippet shows how to do the transformation:
field_ud_datetime/value:
plugin: format_date
source: src_datetime
from_format: 'Y/m/j H:i:s'
to_format: 'Y-m-d\TH:i:s'
from_timezone: 'America/Managua'
to_timezone: 'UTC'If you need the timezone to be dynamic, things get a bit harder. The from_timezone and to_timezone settings expect a literal value. It is not possible to read a source column to set these configurations. An alternative is that your source column includes timezone information like 2019/12/24 19:15:30 -07:00. In that case, you would need to tweak the from_format to include the timezone component and leave out the from_timezone configuration.
Things to consider
Date migrations can be tricky because they can be affected by things outside of the Migrate API. Here is a non-exhaustive list of things to consider:
- For date and time fields, the transformation might be affected by your server’s timezone if you do not manually set the
from_timezoneconfiguration. - People might see the date and time according to the preferences in their user profile. That is, two users might see a different value for the same migrated field if their preferred timezones are not the same.
- For date only fields, the user might see a time depending on the format used to display them. A list of available formats can be found at
/admin/config/regional/date-time. - A field can always be configured to be presented in a specific timezone. This would override the site’s timezone and the user’s preferred timezone.
What did you learn in today’s blog post? Did you know that entity properties and date fields expect different destination formats? Did you know how to do timezone conversions? What challenges have you found when migrating dates and times? Please share your answers in the comments. Also, I would be grateful if you shared this blog post with others.
Next: Migrating addresses into Drupal
This blog post series, cross-posted at UnderstandDrupal.com as well as here on Agaric.coop, is made possible thanks to these generous sponsors. Contact Understand Drupal if your organization would like to support this documentation project, whether it is the migration series or other topics.
David has been doing web development for 12 years now and Drupal development for the last 6. In his early days as programmer he read about the Free Software movement and immediately wanted to be part of it.
He started contributing to different free software projects but the majority of his work has been made for Drupal, as a core contributor and helping to fix and improve some contrib modules he enjoys helping Drupal become better so that more people can adopt it.
Agaric hosts a weekly online gathering known as Show and Tell. Participants share tips and tricks we have learned and pose questions to other developers on tasks or projects we are working on. Each week we ask people to send us a little info on what they would like to present. This is not a prerequisite, just a suggestion. Having advance notice of presentations allows us to get the word out to others that may be interested, but you can just show up, and there will most likely be time to present for 5-10 minutes. Sign onto the Show and Tell mailing list and be notified of upcoming Show and Tell events.
Recently we have opened up the Show and Tell chat to bond with other cooperatives that do web development work. Agaric was contacted by members of Fiqus.coop in Argentina as they had started an initiative to meet other cooperative developers and share values and goals. No one had sent notice of a presentation, so we switched the topic of the chat to be more of a meet and greet to get to know each other better with the goal in mind to be able to share our work on projects. The value of the meeting was immediately apparent as we delved into conversation with a few members of Fiqus.
Next, we invited more developers to take part in the discussion, and the doors were opened to share more deeply and connect. This week our meeting was over the top! Nicolas Dimarco led us through a short presentation of slides that revealed a Federated process and workflow to share development with members of multiple cooperatives. The plan is so simple that everyone immediately understood and the conversation that ensued was compelling, and the questions were indicative of where we need to educate each other about cooperative principles vs. corporate tactics. We need more discussion on trust and friendship. There are so many developers in corporate jobs that have asked me how a web development cooperative works and how does a project run without a manager. I first explain that projects do have managers, but they are managing the work, not the people. Taking time to get to know each other's skills and passions about programming is a core part of being able to work together in a Federation. Fiqus.coop has made it plain and simple for all to see the path to sharing work on projects!
Here is a link to the video recording of the chat where Nicolas Dimarco of Fiqus.coop presents the formula for federated work among cooperatives. Here is a link to the notes from the meeting on 3/20/2019 and some past Show and Tell meetings.
More information on Show and Tell.
Some Drupal shops already work together on projects and we can help that grow by sharing our experiences. We would love to hear about the ways you work and the processes you have discovered that make sharing work on projects a success!
Teachers with GUTS
Empowering educators to teach science confidently.
Here is how to deal with the surprising-to-impossible-seeming error "Unable to uninstall the MySQL module because: The module 'MySQL' is providing the database driver 'mysql'.."
Like, why is it trying to uninstall anything when you are installing? Well, it is because you are installing with existing configuration— and your configuration is out-of-date. This same problem will happen on configuration import on a Drupal website, too. (See update below for those steps!)
Really this error message is a strong reminder to always run database updates and then commit any resulting configuration changes after updating Drupal core or module code.
And so the solution is to roll back the code to Drupal 9.3, do your installation from configuration, and then run the database updates, export configuration, and commit the result.
For example:
git checkout <commit-hash-of-earlier-composer-lock>
composer install
drush -y site:install drutopia --existing-config
git checkout main
composer install
drush -y updb
drush -y cex
git status # Review what is here; git add -p can also help
git add config/
git commit -m "Apply configuration updates from Drupal 9.4 upgrade"
The system update enable_provider_database_driver is the post-update hook that is doing the work here to "Enable the modules that are providing the listed database drivers." Pretty cool feature and a strong reminder to always, always run database updates and commit any configuration changes immediately after any code updates!
Update: Steps on for resolving the unable to uninstall MySQL module error on configuration import
This is what you probably already did, before the drush -y cim failed (luckily, luckily it failed).
composer update
drush -y updb
All that is great! Now continue, not with a config import, but with a config export:
drush -y cex
git status # Review what is here; git add -p can also help
git add config/
git commit -m "Apply configuration updates from Drupal 9.4 upgrade"
Remember, after every composer update and database update, you need to do a configuration export and commit the results— database updates can change configuration, and if you do not commit those, you will undo these intentional and potentially important changes on a configuration import. If you ran into this problem on a configuration import, it is a sign of breakdown in discipline in following these steps!
Every time you bring in code changes with composer update all this must be part and parcel:
composer update
drush -y updb
drush -y cex
git status # Review what is here; git add -p can also help
git add config/
git commit -m "Apply configuration from database updates"
Sign up to be notified when Agaric gives a migration training:
Agaric is excited to announce online training on Drupal migrations and upgrades. In July 2020, we will offer three trainings: Drupal 8/9 content migrations, Upgrading to Drupal 8/9 using the Migrate API, and Getting started with Drupal 9.
We have been providing training for years at Drupal events and privately for clients. At DrupalCon Seattle 2019, our migration training was sold out with 40+ attendees and received very positive feedback. We were scheduled to present two trainings at DrupalCon Minneapolis 2020: one on Drupal migrations and the other on Drupal upgrades. When the conference pivoted to an online event, all trainings were cancelled. To fill the void, we are moving the full training experience online for individuals and organizations who want to learn how to plan and execute successful Drupal migration/upgrade projects.
Up to date with Drupal 9
Drupal is always evolving and the Migrate API is no exception. New features and improvements are added all the time. We regularly update our curriculum to cover the latest changes in the API. This time, both trainings will use Drupal 9 for all the examples! If you are still using Drupal 8, don't worry as the example code is compatible with both major versions of Drupal. We will also cover the differences between Drupal 8 and 9.
Drupal 8/9 content migrations
In this training you will learn to move content into Drupal 8 and 9 using the Migrate API. An overview of the Extract-Transform-Load (ETL) pattern that migrate implements will be presented. Source, process, and destination plugins will be explained to show how each affects the migration process. By the end of the workshop, you will have a better understanding on how the migrate ecosystem works and the thought process required to plan and perform migrations. All examples will use YAML files to configure migrations. No PHP coding required.
Date: Tuesday, July 21, 2020
Time: 9 AM – 5 PM Eastern time
Cost: $500 USD
Upgrading to Drupal 8/9 using the Migrate API
In this training you will learn to use the Migrate API to upgrade your Drupal 6/7 site to Drupal 8/9. You will practice different migration strategies, accommodate changes in site architecture, get tips on troubleshooting issues, and much more. After the training, you will know how to plan and execute successful upgrade projects.
Date: Thursday, July 23, 2020
Time: 9 AM – 5 PM Eastern time
Cost: $500 USD
Getting started with Drupal 9
We are also offering a training for people who want to get a solid foundation in Drupal site building. Basic concepts will be explained and put into practice through various exercises. The objective is that someone, who might not even know about Drupal, can understand the different concepts and building blocks to create a website. A simple, fully functional website will be built over the course of the day-long class.
Date: Monday, July 13, 2020
Time: 9 AM – 5 PM Eastern time
Cost: $250 USD
Discounts and scholarships available
Anyone is eligible for a 15% discount on their second training. Additionally, if you are a member of an under-represented community who cannot afford the full price of the training, we have larger discounts and full scholarship available. Ask Agaric to learn more about them.
Customized training available
We also offer customized training for you or your team's specific needs. Site building, module development, theming, and data migration are some of the topics we cover. Check out our training page or ask Agaric for more details. Custom training can be delivered online or on-site in English or Spanish.
Meet your lead trainer
Mauricio Dinarte is a frequent speaker and trainer at conferences around the world. He is passionate about Drupal, teaching, and traveling. Over the last few years, he has presented 30+ sessions and full-day trainings at 20+ DrupalCamps and DrupalCons over America and Europe. In August 2019, he wrote an article every day to share his expertise on Drupal migrations.
We look forward to seeing you online in July at any or all of these trainings!
Micky was a keynote speaker at UMASS Amherst during the NERD Summit event and the closing keynote speaker at LibrePlanet 2019 @M.I.T. She spoke about how we, as people and as programmers, can work our way out of the digital world of Nineteen Eight-Four that we are living in. Rather than having about ten slides of fine print and links in the presentation, we are posting resources in this blog post.
Here is a short-enough-to-write-on-a-business-card link for this page – agaric.coop/libreplanet2019 – for sharing these resources with others more easily.
We value learning new things and helping one another, so every Thursday at 3pm Eastern Time we take an hour to share things we are working on. We have deep conversations on the ways we can work together We use screen-sharing to show projects we are involved in and sometimes we doodle on the white board as we talk.
Everyone is welcome to join the chat or just listen. The informal atmosphere supports us getting to know each other better and form stronger relationships. Anyone can suggest a topic for discussion or give a presentation or ask for input and help on a project. We use a poll to determine if we want to record the session. Agaric hosts these show and tells publicly because we realize some of us work alone or in organizations that do not encourage skill-sharing, or may just be interested in broadening their knowledge or sharing some code. So we invite you—our partners, students, colleagues, friends—to take part in watching or giving short presentations.
Direct link to the Show and Tell chatroom
Get on the Show and Tell Mailing List to receive invitations each week with the upcoming topics.
Do you want to host a Show and Tell discussion or presentation? Here is the email template to send a notice to the list at: showandtell@lists.mayfirst.org If you are already signed up on the email list, you should be able to send a notice to the group! Feel free to choose an alternative time if Thursdays at 3PM ET does not work well for you - experiment!
We shall see you soon!
Louis has been a Linux user since his childhood, although there was a period when he did not want to be free because it was too tricky to get PC games set up in the liberated zone.
As an adult, after deciding on a skill that could pay the rent and leave a little extra Louis bit into Jennifer Robbins' Learning Web Design: A Beginner's Guide to HTML, CSS, JavaScript, and Web Graphics and Marijn Haverbeke's Eloquent Javascript. When he was starting off Louis loved spending non shelf-stocking, fruit cutting, floor mopping, hours solving (or attempting to solve) code challenges. He would spend hours wrestling with problems which he should have given up on much earlier and just learned from the solution.
Professionally, Louis got started working with NOVA Web Development setting up LibreOrganize sites an association management system built with Django. Now with Agaric Louis works developing and configuring Drutopia sites.
Louis is thankful and excited to be the newest member of Agaric. Louis likes worker-coops and exploring the role they play in transitioning to the society of the emancipated worker.
