Wow, that was fun. The very first event for the Boston/Cambridge Worker Ownership Meetup group on October 10th, 2014, was a great success. We met at The Industry Lab, Agaric's new co-working headquarters and event location. We had a great discussion that was led by our friend Gonzalo Chomon of EvoVelo, a cooperative located in Spain.

Gonzalo Chomon shared a wealth of knowledge about the inner workings of cooperatives that engaged everyone from the start. About a dozen attendees talked about the different ways cooperative companies are structured, laws governing different types of companies in Spain and the US and about the different internal processes used to add or remove members from the group.

Attendees were very engaged in the discussion from the start. One member asked about the company structure, how does it work, are there contracts, bylaws? A cooperative company can be structured in any way the partners agree upon. Usually the company is formed as an LLC between the partners with equal shares, but that is not an absolute. An organization the size of Mondragon has an entirely different structure, it is a federation of cooperatives.

The pizza from The Just Crust rocked and was totally consumed by the end of the evening. Many thanks, to our friend Kay Van Valkenburg who brought some lush fruits and scrumptious chocolate chip and dark chocolate dipped cookies—oooooh. Thanks to everyone that came to the first Boston/Cambridge Worker Ownership Meetup at the Industry Lab in Cambridge, despite the brilliant New England weather.

We have already planned our next meetup for November 17th at the Industry Lab.

We will talk with Carolyn Edsell-Vetter, from A Yard & A Half Landscaping Cooperative, based right in Waltham, MA, about transitioning to a coop from another ownership/governance structure. Meet some local people that are currently working in successful worker cooperatives in any industry, doing what they love to do. Let's discuss how they started and how they continue to thrive in this economy. What are the benefits and what are the hurdles to overcome when forming a cooperative relationship with your partners?

Join us! Visit the Boston/Cambridge Worker Ownership Meetup, we have started to list some great coop resources and links, and if you are in the Boston area, let us know.

Find It makes it easier for caretakers to find opportunities for their children online so that they can stay focused on taking care of themselves and their children.

 We look forward to being in communication, perchance to work together or just to meet in person sometime!

The Global Training Days is an initiative coordinated by the Drupal Association to introduce new people to the wonderful world of Drupal. The initiative is about providing free or low-cost training to facilitate people gaining familiarity with Drupal in just a day. Depending on the format, attendees can leave the workshop with a website already built. In this blog post, I will explain how to organize one and share experiences and lessons learned after organizing them in Nicaragua since 2014. It is my hope to inspire you to be part of the initiative and organize a workshop in your local community.

The basics

To get started, find out the next dates when the workshops would be given worldwide. They happen two days in a row (on Friday and Saturday) approximately every 3 months. Dates are set in the last quarter of the previous year and can be checked at https://www.drupal.org/global-training-days.

For the workshop itself there are two options to choose from in terms of time and curriculum. The first is a What is Drupal? half-day workshop where attendees learn the basics of Drupal. If you go this route it is recommended to spend most of the time explaining the different Drupal concepts, and how they relate to each other in order to build a site. Hands-on instructions where each attendee builds a site can be left out due to time constraints, but it is fine to have a demo where you show how to assemble a website using Drupal.

The second option is an Intro to Drupal full-day workshop where attendees learn the basic concepts and have the opportunity to build a simple yet functional Drupal site. For this type of workshop, you can spend the first half explaining concepts and the second half building the site. Remember the introductory nature of the workshop. Do not try to cover advanced site building concepts. Stick to Drupal core and build your site using using nodes, blocks, views, and core modules like Contact. In either version of the workshop, do not go over how to build themes or modules. If you try to cover too many topics, you might not develop them all appropriately. If you really want to cover these topics, it is better to organize a follow up workshop where they could be explained with more time.

Once you decide the date and type of workshop you want to present, sign up at this page. After doing so you will get an email from the Drupal Association. You will also be added to the list of organizers.

Plan ahead

The dates for the workshops are announced way ahead of time. Be an early bird and do not leave things to the last minute. Ask yourself these questions and plan accordingly:

• Am I participating the two days (Friday and Saturday) or only one day?
• Am I presenting a half-day or a full-day workshop?
• Am I providing computers to attendees or they are required to bring a laptop?
• Am I providing lunch for attendees or they are required to bring/buy their own food?
• How many people can attend based on the facility’s capacity?
• How many people can attend based on the number of speakers?
• Does each speaker know which section of the workshop they are going to present?
• If the workshop includes building a Drupal site, which tools will I use?
• Where should I advertise the workshop?
• What is the contingency plan in case the Internet service does not work?

Before the training

It may seem obvious, but make sure you have a place with the proper conditions to give a workshop. You will need:

• a projector
• a whiteboard or chalkboard, or flipchart and markers
• a decent Internet connection
• good room lighting
• comfortable room temperature
• zero sound pollution

The last item is very important as you do not want to yell to be heard. It happened to us once that the room next to the workshop laboratory was occupied by a group of loud children. This not only distracted the attendees, but also forced me to raise my voice to be heard. After speaking loudly for a couple of hours my voice had disappeared.

Even without noise pollution, giving back-to-back trainings two days in a row poses quite a challenge to keeping your voice audible. Make sure you have multiple speakers and take turns presenting. Use a microphone if available. It helps to preserve your voice, and helps participants who might not hear very well.

Once you have the venue figured out, be realistic about how many people you can effectively manage. Organizing a workshop takes lots of effort. It is fine to try to teach as many people as possible, but quality is preferred over quantity. In our experience, teaching small groups is more effective because you can dedicate more time to answer all questions each attendee might have and provide a personalized learning experience.

Make sure you get a confirmation of the venue at least 3 weeks before the training so you have time to promote the workshop. In Nicaragua, we always make the announcement in our official Drupal group and on the community’s Twitter and Facebook accounts. But experience has shown us that only a minority of the attendees find out about the event through these channels. In our case, about 90% of attendees discovered the event after seeing an invitation posted to a non-Drupal specific Facebook group of Nicaraguan developers. The 10% remaining is a combination of word of mouth (past participants recommending the workshop to others), people I meet in various tech meetups, and the channels mentioned before.

When deciding where to advertise the workshop, find the right pond and remember to fish where the fish are. If you have some spare dollars, feel free to invest in paid social media advertisement. No matter where you decide to promote the workshop, include the following information in the announcement:

• Date and times.
• Name of venue and address.
• A photo of the building where the workshop will take place! We have experienced people being unable to find the venue. So, play it safe, be verbose with directions, and add a photo of the venue.
• Whether or not it is required to bring a laptop.
• Whether or not lunch will be provided.
• If registration is required, the url where people can register.
• A note that no previous knowledge is required and that anyone (not only tech people) can join! :-D

Prepare the tools you will use in the training. In Nicaragua, we prefer to use Patrick Drotleff‘s amazing service https://simplytest.me This allows us to focus on building the Drupal site, instead of configuring the myriad of local environments that attendees may have. In our experience, configuring local environments on each attendee’s computer takes too much time. For example, most of the participants that come to our workshops use Windows. Trying to set up a LAMP environment could be surprisingly complicated in this operating system. Even out of the box solutions like XAMPP might not be able to start the web server due to port conflicts. Skype is a recurrent culprit. Starting a MySQL server might also fail. If that happens, skip it altogether and install using SQLite. More often than not, we go the fastest route and use disposable Drupal installations. Keep in mind that you are teaching Drupal site building, not server configuration.

"Anything that can go wrong, will go wrong." Murphy’s law will manifest more often than you would like. Be prepared and have a contingency plan for as much as you can. During one of our workshops, our favorite tool https://simplytest.me was down, but we were prepared with packages for local installation. In another instance, the venue was changing Internet provider and we had no Internet at all. Again, local installations came to the rescue. In such cases, save time by asking people to work in pairs or small groups so you do not have to configure a lot of computers. Bring a copy of your slide deck in a thumb drive so you can present offline in case it is needed. In Nicaragua, not only the tech is community vibrant. Volcanoes are too! With a handful of active volcanoes that produce tremors very often, we need to make sure to have clear instructions on how to leave the facility if necessary. Do your homework and be ready in case anything goes wrong.

On the day of the training

Take with you a laptop, a VGA/DVI/HDMI cable and adapter, and a back up projector if available. Have a copy of your slide deck for offline presenting and all the tools you might need in case the Internet decides to visit Saturn. Arrive early at the venue, set up your computer and projector for the presentation, and write the wi-fi password on the board. If you are providing computers, make sure any tool you intend to use is installed and working.

Take lots of photos (with participants' permission) and share them on social media. Be sure to record the training for posterity. This is not to show off, but to inspire others to organize workshops in their local communities. Make sure to use the official #DrupalGTD hashtag when sharing content. And please respect those who might not want to appear online! In our trainings, after finishing presenting theory and before starting to build sites we ask the attendees for a group photo which we later share on Facebook and Twitter.

We drupalers are fond of freebies. We enjoy stickers, T-shirts, trial online subscriptions, or anything that comes in an event as swag, right? Let’s share this passion with attendees. In our trainings, we use to have a table with lots of stickers for people to take from. In one occasion we also gave away T-shirts. And we have gotten in contact with online education providers to give attendees free trial access to their content. Chris Shattuck from Build a Module has been so kind in offering free access to his amazing library of content for 8 days to every attendee of our workshops. He also gave us some 1-month free memberships to raffle off. And the friendly folks from Knp University provided free access to Drupal-related material to attendees of the November 2015 edition of the workshop. When organizing a workshop make sure to have some treats for your attendees and partner with education providers so they can keep learning when the day is over. Also, show attendees how to stay connected with the broader community by helping them create a drupal.org profile and teaching them how to get involved.

A final comment for this section has to do with attendance. Do not feel bad if the number of people who show up is not the same that the number of people who registered. In fact, almost every time those numbers will not match. Sometimes you will have many attendees and in other occasions only a few. Do not let this disappoint you. Remember what motivated you in the first place to organize the event. Share your knowledge with others, no matter the number of attendees. Personally, I would present any workshop or session as long as at least one person shows up. Everyone’s time is valuable and each attendee sets aside time to listen to you. That is an honor I take with great responsibility, and you should too.

After the training

After you have enjoyed the Global Training Days experience, give thanks to everyone who made the workshop possible. This includes the attendees, co-presenters, venue providers, sponsors, and educational partners. In Nicaragua we post pictures and thank-you notes on Twitter and Facebook. And in those thank-you messages we also include the next dates of the workshop for those who missed it this time.

Of utmost importance is to perform an evaluation of the workshop. Meditate on things that went well, but also on things that went wrong and can be improved. At the end of our workshops we ask for feedback from the attendees. We also take time to improve the slide deck based on the concepts that attendees had difficulty understanding. This often translates to changing the copy, images, and examples used to explain the different concepts.

After all the hard work, give yourself a pat on the back, and share your story with others. You can write a blog post like this or simply post pictures of the workshop online. The Drupal Association is also interested in feedback from you and the attendees, so do not hesitate in providing it.

My wish list

The Drupal Association provides a lot of support and material to promote these events. Unfortunately, this content is only available in English at the moment. I would love to see this content translated to every language spoken by a member of our community. In Spanish for example, there is a great community that could help with the task of translating the content. The same can be said for communities which speak other languages. Eduardo Garcia is running for Director at Large within the Drupal Association and he has interesting ideas about making our community more inclusive for non-English speaking members.

Another wish of mine is collaboration in curriculum building. When the Nicaraguan community started to participate in the Global Trainings Days initiative back in 2014, we found little content that could be used in the workshop. The Drupal Association offers a curriculum in case you need it, but it is in English and our audience speaks Spanish. Because of this, we basically had to start creating a curriculum from scratch. The curriculum we use in Nicaragua is far from perfect, but the content has improved a lot over time. This has taken a lot of time and effort, yet there is no need to repeat the process again. Because of this, we have made our content available under a Creative Commons Attribution - Non Commercial - Share Alike license for anyone to use and adapt to their needs. So far we have available both a slide deck and a video recording of the theory part of one of the workshops. Unfortunately, some videos do not have audio, but we are planning on making another recording in a future edition. Also, I have started working on a written version of the workshop. This document is in a very early stage, but I will continue adding to it as time permits. It would be great if other communities would use this documentation and provide feedback to improve it. Collaborating in building a curriculum in different languages would be amazing. If you like this idea, start working on the curriculum in a new language and promote it to get feedback from others.

Measuring success

There are various parameters that you could use to measure the success of your workshops. For example, the number of attendees and the number of people who keep engaged with the local community thereafter. In Nicaragua, we have organized 10 workshops since 2014 and over 120 people have attended. Some keep engaged, while others do not. But numbers can be cold, so we measure success in non-quantitative ways. For example, seeing people smile when they understand the basic concepts of Drupal, or viewing their excitement after being able to build a website during the training.

Sometimes, we are able to make a more significant impact on the life of attendees. For instance, we are aware of people who could land a Drupal job after attending the workshop and continuing to learn on their own. The fact that we are making an impact on their lives is our best reward. Here are two testimonials of people who got a job after attending our workshops.

Ada Hernández - Translation and code contributor:

Hello everyone. My name is Ada Hernández Acosta and I am from León, Nicaragua. Attending a Drupal workshop presented by Lucas Hedding and Mauricio Dinarte set a milestone in my professional and personal life. At the workshop I learned the basic of Drupal: nodes, taxonomies, views, modules, themes, and more. I also learned the benefits of using Drupal including its security and no cost for usage. Moreover, a worldwide community is actively working on creating new modules, themes, and security updates. After attending the workshop I developed a strong interest in Drupal and decided let Drupal take part in my life. Today I work at MTech, LLC building websites on Drupal. Now I am also a member of this great community and I am able to make contributions to modules I use on the sites I work on. Thank you very much.

Edys Meza - Code contributor:

It was my first experience in something of this nature. I had just finished college and I wanted to learn new technologies. The Global Training Days workshop was a great opportunity for this. After attending the workshop I got an overview of Drupal that helped me a lot in the future. I learned basic concepts (nodes, taxonomies, users, blocks, etc.) each with real life examples; this was intuitive and easy to understand. That day I became part of the great Drupal community. Attending the workshop changed my life. After it I kept learning in the free classes presented by MTech, LLC where I currently work and continue learning everyday. I think these workshops are what make the Drupal community grow and help improve the lives of attendees.

Final comments

Be flexible. Nothing related to the Global Training Day is set in stone. For example, if for any reason you cannot give the training on the day suggested by the Drupal Association, pick another day. It can be in the same week or the week before or after. A good reason to do this is to let yourself rest and have enough time to recover your voice between multiple workshops in the same edition of the initiative.

Community first, companies later. Although the Drupal Association primarily has invited companies to give workshops to promote themselves, growing the Drupal community and sharing your knowledge should be the focus. In Nicaragua, we started to give the training as a local community initiative. It was Drupal Nicaragua, not a company who started giving the workshop in 2014. Later, both Agaric and MTech sponsored the events and education providers partnered with us. Of course, we are very grateful to them for their ongoing support!

Keep calm and enjoy the experience. Undoubtedly there will be mistakes, particularly the first few times you organize the workshop. In Nicaragua, we have given 10 workshops so far and none has been perfect. What is important is that you learn from the mistakes and improve for the next edition. Sometimes we face many difficulties, but we are able to work through them. At the end of the workshop, you will be rewarded by the smiles of people who have learned to build a simple yet functional website in a matter of hours. :-D

This is how the Nicaraguan Drupal community organizes Global Training Days. It is my hope that after reading through you feel inspired to give a training in your local community. When you are ready to take the challenge, pour your heart into it. Upload some photos of the workshop once it is finished, and share your experience, so that the entire Drupal community can benefit from it. If you have already participated in the Global Training Days, please share your story in the comments below. Have fun!

Special thanks to Alina Mackenzie and Lizz Trudeau for their thorough reviews and suggestions to this blog post. Also to Lucas Hedding who has helped move the Nicaraguan Drupal community forward by creating documentation and organizing events. His contributions to the local community are invaluable. And thanks to the amazing BADCamp organizers for providing me financial support to attend the camp the last two years. I have learned a lot during these events and the acquired knowledge has greatly improved the curriculum used at the Global Training Days workshop. The effort that you pour into the camp and keeping it free is inspiring and greatly appreciated. Keep up the good work! :-)

Sign up to be notified when Agaric gives a migration training:

Agaric builds tools for medical and scientific communities to advance their work, enhance collaboration, and improve outcomes.

Agaric is grateful to the Drupal community for all the effort poured into the amazing collaborative project. As part of giving back to it, we go to conferences to share with others what we have learned. These are some events where Agaric will be presenting this month.

Eastern Conference for Workplace Democracy

This is a convergence of worker-owned cooperatives. Representatives come from all over the country to attend workshops and sessions on all things related to owning a cooperative. It will be help in New York City on weekend of June 9th -11th at the John Jay College of Criminal Justice.

Benjamin and Micky will be hosting a workshop/discussion with Danny Spitzberg on Drutopia. They will cover how it can help cooperatives and smaller businesses have a we presence above and beyond the costs they can afford by consolidating the hosting and feature development into a group effort.

Montreal Drupal Camp

This event will take place on June 15-18 at John Molson School of Business de l'Université Concordia. Benjamin will be speaking on how Software as a Service can lead to long-term success in a software project.

Twin Cities Drupal Camp in Minneapolis

At Twin Cities Agaric will be presenting one workshop and two sessions.

On Thursday, June 22, Benjamin and Mauricio will be presenting the Getting Started with Drupal workshop. It is aimed to people who are just starting with Drupal and want to have a birds eye view of how the system works. As part of the workshop attendees will have the chance to create a simple yet functional website to put in practice their new knowledge. The organizers have gone above and beyond to make this training FREE for everyone! You do not even need a camp ticket to participate. You just need to register.

On Saturday, June 24, Mauricio will be presenting on Drupal 8 Twig recipes. This will be an overview of the theme system in Drupal 8 and will include practical example of modifying the default markup to your needs. The same day, Benjamin will present his Software as a Service.

Design4Drupal

This is THE yearly Camp for Drupal doers in Boston and it happens June 22nd-23rd. Micky will be hosting a workshop/discussion on Drutopia, an initiative within the Drupal project based in social justice values and focused on building collectively owned online tools. Current focuses include two Drupal distributions aimed at grassroots groups also offered as software as a service, ensuring that the latest technology is accessible to low-resourced communities.
Agaric will have a busy month attending and speaking at conferences. Please come to say hi and have fun with us.

In recent posts we have explored the Migrate Plus and Migrate Tools modules. They extend the Migrate API to provide migrations defined as configuration entities, groups to share configuration among migrations, a user interface to execute migrations, among other things. Yet another benefit of using Migrate Plus is the option to leverage the many process plugins it provides. Today, we are going to learn about two of them: `entity_lookup` and `entity_generate`. We are going to compare them with the `migration_lookup` plugin, show how to configure them, and explain their compromises and limitations. Let’s get started.

What is the difference among the migration_lookup, entity_lookup, entity_generate plugins?

In the article about migration dependencies we covered the `migration_lookup` plugin provided by the core Migrate API. It lets you maintain relationships among entities that are being imported. For example, if you are migrating a node that has associated users, taxonomy terms, images, paragraphs, etc. This plugin has a very important restriction: the related entities must come from another migration. But what can you do if you need to reference entities that already exists system? You might already have users in Drupal that you want to assign as node authors. In that case, the `migration_lookup` plugin cannot be used, but `entity_lookup` can do the job.

The `entity_lookup` plugin is provided by the Migrate Plus module. You can use it to query any entity in the system and get its unique identifier. This is often used to populate entity reference fields, but it can be used to set any field or property in the destination. For example, you can query existing users and assign the `uid` node property which indicates who created the node. If no entity is found, the module returns a `NULL` value which you can use in combination of other plugins to provide a fallback behavior. The advantage of this plugin is that it does not require another migration. You can query any entity in the entire system.

The `entity_generate` plugin, also provided by the Migrate Plus module, is an extension of `entity_lookup`. If no entity is found, this plugin will automatically create one. For example, you might have a list of taxonomy terms to associate with a node. If some of the terms do not exist, you would like to create and relate them to the node.

Note: The `migration_lookup` offers a feature called stubbing that neither `entity_lookup` nor `entity_generate` provides. It allows you to create a placeholder entity that will be updated later in the migration process. For example, in a hierarchical taxonomy terms migration, it is possible that a term is migrated before its parent. In that case, a stub for the parent will be created and later updated with the real data.

Getting the example code

You can get the full code example at https://github.com/dinarcon/ud_migrations The module to enable is `UD Config entity_lookup and entity_generate examples` whose machine name is `ud_migrations_config_entity_lookup_entity_generate`. It comes with one JSON migrations: `udm_config_entity_lookup_entity_generate_node`. Read this article for details on migrating from JSON files. The following snippet shows a sample of the file:

{
  "data": {
    "udm_nodes": [
      {
        "unique_id": 1,
        "thoughtful_title": "Amazing recipe",
        "creative_author": "udm_user",
        "fruit_list": "Apple, Pear, Banana"
      },
      {...},
      {...},
      {...}
    ]
  }
}

Additionally, the example module creates three users upon installation: 'udm_user', 'udm_usuario', and 'udm_utilisateur'. They are deleted automatically when the module is uninstalled. They will be used to assign the node authors. The example will create nodes of types "Article" from the standard installation profile. You can execute the migration from the interface provided by Migrate Tools at `/admin/structure/migrate/manage/default/migrations`.

Using the entity_lookup to assign the node author

Let’s start by assigning the node author. The following snippet shows how to configure the `entity_lookup` plugin to assign the node author:

uid:
  - plugin: entity_lookup
    entity_type: user
    value_key: name
    source: src_creative_author
  - plugin: default_value
    default_value: 1

The `uid` node property is used to assign the node author. It expects an integer value representing a user ID (`uid`). The source data contains usernames so we need to query the database to get the corresponding user IDs. The users that will be referenced were not imported using the Migrate API. They were already in the system. Therefore, `migration_lookup` cannot be used, but `entity_lookup` can.

The plugin is configured using three keys. `entity_type` is set to machine name of the entity to query: `user` in this case. `value_key` is the name of the entity property to lookup. In Drupal, the usernames are stored in a property called `name`. Finally, `source` specifies which field from the source contains the lookup value for the `name` entity property. For example, the first record has a `src_creative_author` value of `udm_user`. So, this plugin will instruct Drupal to search among all the users in the system one whose `name` (username) is `udm_user`. If a value if found, the plugin will return the user ID. Because the `uid` node property expects a user ID, the return value of this plugin can be used directly to assign its value.

What happens if the plugin does not find an entity matching the conditions? It returns a `NULL` value. Then it is up to you to decide what to do. If you let the `NULL` value pass through, Drupal will take some default behavior. In the case of the `uid` property, if the received value is not valid, the node creation will be attributed to the anonymous user (uid: 0). Alternatively, you can detect if `NULL` is returned and take some action. In the example, the second record specifies the "udm_not_found" user which does not exists. To accommodate for this, a process pipeline is defined to manually specify a user if `entity_lookup` did not find one. The `default_value` plugin is used to return `1` in that case. The number represents a user ID, not a username. Particularly, this is the user ID of "super user" created when Drupal was first installed. If you need to assign a different user, but the user ID is unknown, you can create a pseudofield and use the `entity_lookup` plugin again to finds its user ID. Then, use that pseudofield as the default value.

Important: User entities do not have bundles. Do not set the `bundle_key` nor `bundle` configuration options of the `entity_lookup`. Otherwise, you will get the following error: "The entity_lookup plugin found no bundle but destination entity requires one." Files do not have bundles either. For entities that have bundles like nodes and taxonomy terms, those options need to be set in the `entity_lookup` plugin.

Using the entity_generate to assign and create taxonomy terms

Now, let’s migrate a comma separated list of taxonomy terms. An example value is `Apple, Pear, Banana`.  The following snippet shows how to configure the `entity_generate` plugin to look up taxonomy terms and create them on the fly if they do not exist:

field_tags:
  - plugin: skip_on_empty
    source: src_fruit_list
    method: process
    message: 'No src_fruit_list listed.'
  - plugin: explode
    delimiter: ','
  - plugin: callback
    callable: trim
  - plugin: entity_generate
    entity_type: taxonomy_term
    value_key: name
    bundle_key: vid
    bundle: tags

The terms will be assigned to the `field_tags` field using a process pipeline of four plugins:

• `skip_on_empty` will skip the processing of this field if the record does not have a `src_fruit_list` column.
• `explode` will break the string of comma separated files into individual elements.
• `callback` will use the `trim` PHP function to remove any whitespace from the start or end of the taxonomy term name.
• `entity_generate` takes care of finding the taxonomy terms in the system and creating the ones that do not exist.

For a detailed explanation of the `skip_on_empty` and `explode` plugins see this article. For the `callback` plugin see this article. Let’s focus on the `entity_generate` plugin for now. The `field_tags` field expects an array of taxonomy terms IDs (`tid`). The source data contains term names so we need to query the database to get the corresponding term IDs. The taxonomy terms that will be referenced were not imported using the Migrate API. And they might exist in the system yet. If that is the case, they should be created on the fly. Therefore, `migration_lookup` cannot be used, but `entity_generate` can.

The plugin is configured using five keys. `entity_type` is set to machine name of the entity to query: `taxonomy_term` in this case. `value_key` is the name of the entity property to lookup. In Drupal, the taxonomy term names are stored in a property called `name`. Usually, you would include a `source` that specifies which field from the source contains the lookup value for the `name` entity property. In this case it is not necessary to define this configuration option. The lookup value will be passed from the previous plugin in the process pipeline. In this case, the trimmed version of the taxonomy term name.

If, and only if, the entity type has bundles, you also must define two more configuration options: `bundle_key` and `bundle`. Similar to `value_key` and `source`, these extra options will become another condition in the query looking for the entities. `bundle_key` is the name of the entity property that stores which bundle the entity belongs to. `bundle` contains the value of the bundle used to restrict the search. The terminology is a bit confusing, but it boils down to the following. It is possible that the same value exists in multiple bundles of the same entity. So, you must pick one bundle where the lookup operation will be performed. In the case of the taxonomy term entity, the bundles are the vocabularies. Which vocabulary a term belongs to is associated in the `vid` entity property. In the example, that is `tags`. Let’s consider an example term of "Apple". So, this plugin will instruct Drupal to search for a taxonomy term whose `name` (term name) is "Apple" that belongs to the "tags" `vid` (vocabulary).

What happens if the plugin does not find an entity matching the conditions? It will create one on the fly! It will use the value from the source configuration or from the process pipeline. This value will be used to assign the `value_key` entity property for the newly created entity. The entity will be created in the proper bundle as specified by the `bundle_key` and `bundle` configuration options. In the example, the terms will be created in the `tags` vocabulary. It is important to note that values are trimmed to remove whispaces at the start and end of the name. Otherwise, if your source contains spaces after the commas that separate elements, you might end up with terms that seem duplicated like "Apple" and " Apple".

More configuration options

Both `entity_lookup` and `entity_generate` share the previous configuration options. Additionally, the following options are only available:
`ignore_case` contains a boolean value to indicate if the query should be case sensitive or not. It defaults to true.
`access_check` contains a boolean value to indicate if the system should check whether the user has access to the entity. It defaults to true.
`values` and `default_values` apply only to the `entity_generate` plugin. You can use them to set fields that could exist in the destination entity. An example configuration is included in the code for the plugin.

One interesting fact about these plugins is that none of the configuration options is required. The `source` can be skipped if the value comes from the process pipeline. The rest of the configuration options can be inferred by code introspection. This has some restrictions and assumptions. For example, if you are migrating nodes, the code introspection requires the `type` node property defined in the process section. If you do not set one because you define a `default_bundle` in the destination section, an error will be produced. Similarly, for entity reference fields it is assumed they point to one bundle only. Otherwise, the system cannot guess which bundle to lookup and an error will be produced. Therefore, always set the `entity_type` and `value_key` configurations. And for entity types that have bundles, `bundle_key` and `bundle` must be set as well.

Note: There are various open issues contemplating changes to the configuration options. See this issue and the related ones to keep up to date with any future change.

Compromises and limitations

The `entity_lookup` and `entity_generate` plugins violate some ETL principles. For example, they query the destination system from the process section. And in the case of `entity_generate` it even creates entities from the process section. Ideally, each phase of the ETL process is self contained. That being said, there are valid uses cases to use these plugins and they can you save time when their functionality is needed.

An important limitation of the `entity_generate` plugin is that it is not able to clean after itself. That is, if you rollback the migration that calls this plugin, any created entity will remain in the system. This would leave data that is potentially invalid or otherwise never used in Drupal. Those values could leak into the user interface like in autocomplete fields. Ideally, rolling back a migration should delete any data that was created with it.

The recommended way to maintain relationships among entities in a migration project is to have multiple migrations. Then, you use the `migration_lookup` plugin to relate them. Throughout the series, several examples have been presented. For example, this article shows how to do taxonomy term migrations.

What did you learn in today’s blog post? Did you know how to configure these plugins for entities that do not have bundles? Did you know that reverting a migration does not delete entities created by the `entity_generate` plugin? Did you know you can assign fields in the generated entity? Share your answers in the comments. Also, I would be grateful if you shared this blog post with others.

Next: How to debug Drupal migrations - Part 1

This blog post series, cross-posted at UnderstandDrupal.com as well as here on Agaric.coop, is made possible thanks to these generous sponsors. Contact Understand Drupal if your organization would like to support this documentation project, whether it is the migration series or other topics.

With Europe threatening $25,000,000 fines and Facebook losing $80,000,000,000 of stock value, are you paying attention to data privacy yet? If millions and billions of dollars in news headlines never grabbed you, maybe you've noticed the dozens of e-mails from services you'd forgotten ever signing up for, declaring how much they respect your right to control your data. These e-mails are silly and possibly illegal, but they nonetheless welcome us to a better world of greater privacy rights and people's control of their own data that we web developers should embrace.

The huge potential fines (for large companies, the sky's the limit at four percent of global revenue) come from the European Union's General Data Protection Regulation, and they signal that the GDPR is more than a suggestion. If you're not a European-based company, the European Union does not intend to discriminate: You're still liable when citizens of member states use your services or are monitored by you.

Don't lose sleep for Facebook's wealthy stockholders. That sizeable dip in Facebook stock was not due to the impending GDPR enforcement, but came in the wake of the Cambridge Analytica scandal. Since then, the privacy-invading monopoly so many rich people are betting on regained its market cap and then some. (GDPR-related lawsuits are just starting.)

There's a lot of good resources for GDPR-proofing existing sites (see the bottom of this article); the work ranges from trivial for most sites to monumental tasks for web developers who, fortunately for me, aren't me (and who have finished their labor, I hope, as GDPR enforcement took effect today).

The fun and exciting part starts when we get to build new sites or new features on existing sites and from the beginning put privacy by design into practice (which also is in the law). And yes, I'm referring to complying with a continental government's regulations as fun and exciting.

Throughout the series, we explored many migration topics. We started with an overview of the ETL process and workflows for managing migrations. Then, we presented example migrations for different entities: nodes, files, images, taxonomy terms, users, and paragraphs. Next, we shifted focus to migrations from different sources: CSV, JSON, XML, Google Sheet, Microsoft Excel, and LibreOffice Calc files. Later, we explored how to manage migrations as configuration, use groups to share configuration, and execute migrations from the user interface. Finally, we gave recommendations and provided tools for debugging migrations from the command line and the user interface. Although we covered a lot of ground, we only scratched the surface. The Migrate API is so flexible that its use cases are virtually endless. To wrap up the series, we present an introduction to a very popular topic: Drupal upgrades. Let’s get started.

Note: In this article, when we talk about Drupal 7, the same applies to Drupal 6.

What is a Drupal upgrade?

The information we presented in the series is generic enough that it applies to many types of Drupal migrations. There is one particular use case that stands out from the rest: Drupal upgrades. An upgrade is the process of taking your existing Drupal site and copy its configuration and content over to a new major version of Drupal. For example, going from Drupal 6 or 7 to Drupal 8. The following is an oversimplification of the workflow to perform the upgrade process:

• Install a fresh Drupal 8 site.
• Add credentials so that the new site can connect to Drupal 7’s database.
• Use the Migrate API to generate migration definition files. They will copy over Drupal 7’s configuration and content. This step is only about generating the YAML files.
• Execute those migrations to bring the configuration and content over to Drupal 8.

Preparing your migration

Any migration project requires a good plan of action, but this is particularly important for Drupal upgrades. You need to have a general sense of how the upgrade process works, what assumptions are made by the system, and what limitations exist. Read this article for more details on how to prepare a site for upgrading it to Drupal 8. Some highlights include:

• Both sites need to be in the latest stable version of their corresponding branch. That means the latest release of Drupal 7 and 8 at the time of performing the upgrade process. This also applies to any contributed module.
• Do not do any configuration of the Drupal 8 site until the upgrade process is completed. Any configuration you make will be overridden, and there is no need for it anyways. Part of the process includes recreating the old site’s configuration: content types, fields, taxonomy vocabularies, etc.
• Do not create content on the Drupal 8 site until the upgrade process is completed. The upgrade process will keep the unique identifiers from the source site: `nid`, `uid`, `tid`, `fid`, etc. If you were to create content, the references among entities could be broken when the upgrade process overrides the unique identifiers. To prevent data loss, wait until the old site's content has been migrated to start adding content to the new site.
• For the system to detect a module’s configuration to be upgraded automatically, it has to be enabled on both sites. This applies to contributed modules in Drupal 7 (e.g., link) that were moved to core in Drupal 8. Also to Drupal 7 modules (e.g. address field) that were superseded by a different one in Drupal 8 (e.g. address). In any of those cases, as long as the modules are enabled on both ends, their configuration and content will be migrated. This assumes that the Drupal 8 counterpart offers an automatic upgrade path.
• Some modules do not offer automatic upgrade paths. The primary example is the Views module. This means that any view created in Drupal 7 needs to be manually recreated in Drupal 8.
• The upgrade procedure is all about moving data, not logic in custom code. If you have custom modules, the custom code needs to be ported separately. If those modules store data in Drupal’s database, you can use the Migrate API to move it over to the new site.
• Similarly, you will have to recreate the theme from scratch. Drupal 8 introduced Twig which is significantly different to the PHPTemplate engine used by Drupal 7.

Customizing your migration

Note that the creation and execution of the migration files are separate steps. Upgrading to a major version of Drupal is often a good opportunity to introduce changes to the website. For example, you might want to change the content modeling, navigation, user permissions, etc. To accomplish that, you can modify the generated migration files to account for any scenario where the new site’s configuration diverts from the old one. And only when you are done with the customizations, you execute the migrations. Examples of things that could change include:

• Combining or breaking apart content types.
• Moving data about people from node entities to user entities, or vice versa.
• Renaming content types, fields, taxonomy vocabularies and terms, etc.
• Changing field types. For example, going from Address Field module in Drupal 7 to Address module in Drupal 8.
• Merging multiple taxonomy vocabularies into one.
• Changing how your content is structured. For example, going from a monolithic body field to paragraph entities.
• Changing how your multimedia files are stored. For example, going from image fields to media entities.

Performing the upgrade

There are two options to perform the upgrade. In both cases, the process is initiated from the Drupal 8 site. One way is using the Migrate Drupal UI core module to perform the upgrade from the browser’s user interface. When the module is enabled, go to `/upgrade` and provide the database credentials of the Drupal 7 site. Based on the installed modules on both sites, the system will give you a report of what can be automatically upgraded. Consider the limitations explained above. While the upgrade process is running, you will see a stream of messages about the operation. These messages are logged to the database so you can read them after the upgrade is completed. If your dataset is big or there are many expensive operations like password encryption, the process can take too long to complete or fail altogether.

The other way to perform the upgrade procedure is from the command line using Drush. This requires the Migrate Upgrade contributed module. When enabled, it adds Drush commands to import and rollback a full upgrade operation. You can provide database connection details of the old site via command line options. One benefit of using this approach is that you can create the migration files without running them. This lets you do customizations as explained above. When you are done, you can run the migrations following the same workflow of manually created ones.

Known issues and limitations

Depending on whether you are upgrading from Drupal 6 or 7, there is a list of known issues you need to be aware of. Read this article for more information. One area that can be tricky is multilingual support. As of this writing, the upgrade path for multilingual sites is not complete. Limited support is available via the Migrate Drupal Multilingual core module. There are many things to consider when working with multilingual migrations. For example, are you using node or field translations? Do entities have revisions? Read this article for more information.

Upgrade paths for contributed modules

The automatic upgrade procedure only supports Drupal core modules. This includes modules that were added to core in Drupal 8. For any other contributed module, it is the maintainers’ decision to include an automatic upgrade path or not. For example, the Geofield module provides an upgrade path. It is also possible that a module in Drupal 8 offers an upgrade path from a different module in Drupal 7. For example, the Address module provides an upgrade path from the Address Field module. Drupal Commerce also provides some support via the Commerce Migrate module.

Not every module offers an automated upgrade path. In such cases, you can write custom plugins which ideally are contributed back to Drupal.org ;-) Or you can use the techniques learned in the series to transform your source data into the structures expected by Drupal 8. In both cases, having a broad understanding of the Migrate API will be very useful.

Upgrade strategies

There are multiple migration strategies. You might even consider manually recreating the content if there is only a handful of data to move. Or you might decide to use the Migrate API to upgrade part of the site automatically and do a manual copy of a different portion of it. You might want to execute a fully automated upgrade procedure and manually clean up edge cases afterward. Or you might want to customize the migrations to account for those edge cases already. Michael Anello created an insightful presentation on different migration strategies. Our tips for writing migrations apply as well.

Drupal upgrades tend to be fun, challenging projects. The more you know about the Migrate API the easier it will be to complete the project. We enjoyed writing this overview of the Drupal Migrate API. We would love to work on a follow up series focused on Drupal upgrades. If you or your organization could sponsor such endeavor, please reach out to us via the site’s contact form.

What about upgrading to Drupal 9?

In March 2017, project lead Dries Buytaert announced a plan to make Drupal upgrades easier forever. This was reinforced during his keynote at DrupalCon Seattle 2019. You can watch the video recording in this link. In short, Drupal 9.0 will be the latest point release of Drupal 8 minus deprecated APIs. This has very important implications:

• When Drupal 9 is released, the Migrate API should be mostly the same as Drupal 8. Therefore, anything that you learn today will be useful for Drupal 9 as well.
• As long as your code does not use deprecated APIs, upgrading from Drupal 8 to Drupal 9 will be as easy as updating from Drupal 8.7 to 8.8.
• Because of this, there is no need to wait for Drupal 9 to upgrade your Drupal 6 or 7 site. You can upgrade to Drupal 8 today.

Thank you!

And that concludes the #31DaysOfMigration series. For joining us in this learning experience, thank you very much! ¡Muchas gracias! Merci beaucoup! :-D We are also very grateful to Agaric.coop, Drupalize.Me, and Centarro.io for sponsoring this series.

What did you learn in today’s blog post? Did you know the upgrade process is able to copy content and configuration? Did you know that you can execute the upgrade procedure either from the user interface or the command line? Share your answers in the comments. Also, we would be grateful if you shared this blog post with others.

This blog post series, cross-posted at UnderstandDrupal.com as well as here on Agaric.coop, is made possible thanks to these generous sponsors: Drupalize.me by Osio Labs has online tutorials about migrations, among other topics, and Agaric provides migration trainings, among other services.  Contact Understand Drupal if your organization would like to support this documentation project, whether it is the migration series or other topics.