¿Por qué Libre?

Una nota sobre otras palabras: Software Libre o Código Abierto

Otras lecturas

We are excited to tailor private trainings to suit your needs. Here you will get an idea of the types of private trainings we have given before and would love to give again.

Let us know if you have any questions or special requests

Get Started with Drupal

We love to introduce people to Drupal's basics. We offer short introductory seminars for small groups on an on-demand basis for site building, templating, and developing extensions. Each seminar incorporates when appropriate the information architecture, user experience testing, and leadership and collaboration techniques needed to make any project a success.

Build Your Skills

We teach development teams and solo coders in-person or online, with the program tailored to the problems you are trying to solve. We have provided such capacity-building in Drupal programming, theming and templating, data migration, JavaScript frameworks, and more— and we are always upping our technology skills, so let us know what you are looking for and we may be just ahead of you on the learning curve— and ready to throw down a rope ladder, or at least a grappling hook.

In addition, we offer courses with a defined curricula on building, theming, and developing sites. And we give decision-maker seminars for leaders who need a better understanding of the technology underlying their projects.

Tell Us What You Need!

Don't hesitate to get help with your migrations!

The Migrate API is a very flexible and powerful system that allows you to collect data from different locations and store them in Drupal. It is, in fact, a full-blown extract, transform, and load (ETL) framework. For instance, it could produce CSV files. Its primary use is to create Drupal content entities: nodes, users, files, comments, etc. The API is thoroughly documented, and their maintainers are very active in the #migration slack channel for those needing assistance. The use cases for the Migrate API are numerous and vary greatly. Today we are starting a blog post series that will cover different migrate concepts so that you can apply them to your particular project.

Understanding the ETL process

Extract, transform, and load (ETL) is a procedure where data is collected from multiple sources, processed according to business needs, and its result stored for later use. This paradigm is not specific to Drupal. Books and frameworks abound on the topic. Let’s try to understand the general idea by following a real life analogy: baking bread. To make some bread, you need to obtain various ingredients: wheat flour, salt, yeast, etc. (extracting). Then, you need to combine them in a process that involves mixing and baking (transforming). Finally, when the bread is ready, you put it into shelves for display in the bakery (loading). In Drupal, each step is performed by a Migrate plugin:

The extract step is provided by source plugins.
The transform step is provided by process plugins.
The load step is provided by destination plugins.

As it is the case with other systems, Drupal core offers some base functionality which can be extended by contributed modules or custom code. Out of the box, Drupal can connect to SQL databases including previous versions of Drupal. There are contributed modules to read from CSV files, XML documents, JSON and SOAP feeds, WordPress sites, LibreOffice Calc and Microsoft Office Excel files, Google Sheets, and much more.

The list of core process plugins is impressive. You can concatenate strings, explode or implode arrays, format dates, encode URLs, look up already migrated data, among other transform operations. Migrate Plus offers more process plugins for DOM manipulation, string replacement, transliteration, etc.

Drupal core provides destination plugins for content and configuration entities. Most of the time, targets are content entities like nodes, users, taxonomy terms, comments, files, etc. It is also possible to import configuration entities like field and content type definitions. This is often used when upgrading sites from Drupal 6 or 7 to Drupal 8. Via a combination of source, process, and destination plugins, it is possible to write Commerce Product Variations, Paragraphs, and more.

Technical note: The Migrate API defines another plugin type: `id_map`. They are used to map source IDs to destination IDs. This allows the system to keep track of records that have been imported and roll them back if needed.

Drupal migrations: a two step process

Performing a Drupal migration is a two step process: writing the migration definitions and executing them. Migration definitions are written in YAML format. These files contain information about how to fetch data from the source, how to process the data, and how to store it in the destination. It is important to note that each migration file can only specify one source and one destination. That is, you cannot read form a CSV file and a JSON feed using the same migration definition file. Similarly, you cannot write to nodes and users from the same file. However, you can use as many process plugins as needed to convert your data from the format defined in the source to the format expected in the destination.

A typical migration project consists of several migration definition files. Although not required, it is recommended to write one migration file per entity bundle. If you are migrating nodes, that means writing one migration file per content type. The reason is that different content types will have different field configurations. It is easier to write and manage migrations when the destination is homogeneous. In this case, a single content type will have the same fields for all the elements to process in a particular migration. Once all the migration definitions have been written, you need to execute the migrations. The most common way to do this is using the Migrate Tools module, which provides Drush commands and a user interface (UI) to run migrations. Note that the UI for running migrations only detect those that have been defined as configuration entities using the Migrate Plus module. This is a topic we will cover in the future. For now, we are going to stick to Drupal core’s mechanisms of defining migrations. Contributed modules like Migrate Scheduler, Migrate Manifest and Migrate Run offer alternatives for executing migrations.

Next: Writing your first Drupal migration

This blog post series, cross-posted at UnderstandDrupal.com as well as here on Agaric.coop, is made possible thanks to these generous sponsors: Drupalize.me by Osio Labs has online tutorials about migrations, among other topics, and Agaric provides migration trainings, among other services.  Contact Understand Drupal if your organization would like to support this documentation project, whether it is the migration series or other topics.

Drupal's toolbar second level of menu options and dropdown not showing? Look for "Uncaught DOMException: The quota has been exceeded." errors, as viewable in the Firefox web console. If you see them, the problem is likely due to sites sharing a top-level domain—which is likely if you are using a local development environment like DDEV, and you working on way too many sites at once—combined with a pretty bad Firefox bug that will be fixed in the next release.

To quote Nathan Monfils:

1. Everything from your public domain (abc.tld) counts against your quota, even if it is in a seemingly unrelated subdomain (e.g. my-app.example.com and intranet.example.com).
2. The quota is not recomputed properly, requiring a firefox restart after clearing your data on other subdomains

Note this may affect all sorts of applications, not just Drupal, when you have them running on multiple subdomains of the same top-level domain. So this isn't just about local development environments (and i dislike that DDEV shares their own top-level domain across all the instances you are working on, and while it can be changed i've accepted its way of doing things so i'm on the same page with other developers by default).

Sure, closing more tabs and restarting Firefox could (predictably) have fixed this—and a lot else that's wrong with me, according to everyone i know—but why do that when i can open more tabs and learn precisely how broken everything around me really is?

In Drupal 7 it was useful to do things like this: 

   
function mymodule_content() {
  $links[] = l('Google', 'http://www.google.com');
  $links[] = l('Yahoo', 'http://www.yahoo.com');
  return t('Links: !types', array('!types' => implode(', ', $links)));
}
    

In this case, we are using the exclamation mark to pass the $links into our string but unfortunately, Drupal 8 doesn't have this option in the FormattableMarkup::placeholderFormat(), the good news is that even without this there is a way to accomplish the same thing. 

Today we will learn how to migrate addresses into Drupal. We are going to use the field provided by the Address module which depends on the third-party library commerceguys/addressing. When migrating addresses you need to be careful with the data that Drupal expects. The address components can change per country. The way to store those components also varies per country. These and other important consideration will be explained. Let’s get started.

Getting the code

You can get the full code example at https://github.com/dinarcon/ud_migrations The module to enable is UD address whose machine name is ud_migrations_address. The migration to execute is udm_address. Notice that this migration writes to a content type called UD Address and one field: field_ud_address. This content type and field will be created when the module is installed. They will also be removed when the module is uninstalled. The demo module itself depends on the following modules: address and migrate.

Note: Configuration placed in a module’s config/install directory will be copied to Drupal’s active configuration. And if those files have a dependencies/enforced/module key, the configuration will be removed when the listed modules are uninstalled. That is how the content type and fields are automatically created and deleted.

The recommended way to install the Address module is using composer: composer require drupal/address. This will grab the Drupal module and the commerceguys/addressing library that it depends on. If your Drupal site is not composer-based, an alternative is to use the Ludwig module. Read this article if you want to learn more about this option. In the example, it is assumed that the module and its dependency were obtained via composer. Also, keep an eye on the Composer Support in Core Initiative as they make progress.

Source and destination sections

The example will migrate three addresses from the following countries: Nicaragua, Germany, and the United States of America (USA). This makes it possible to show how different countries expect different address data. As usual, for any migration you need to understand the source. The following code snippet shows how the source and destination sections are configured:

source:
  plugin: embedded_data
  data_rows:
    - unique_id: 1
      first_name: 'Michele'
      last_name: 'Metts'
      company: 'Agaric LLC'
      city: 'Boston'
      state: 'MA'
      zip: '02111'
      country: 'US'
    - unique_id: 2
      first_name: 'Stefan'
      last_name: 'Freudenberg'
      company: 'Agaric GmbH'
      city: 'Hamburg'
      state: ''
      zip: '21073'
      country: 'DE'
    - unique_id: 3
      first_name: 'Benjamin'
      last_name: 'Melançon'
      company: 'Agaric SA'
      city: 'Managua'
      state: 'Managua'
      zip: ''
      country: 'NI'
  ids:
    unique_id:
      type: integer
destination:
  plugin: 'entity:node'
  default_bundle: ud_address

Note that not every address component is set for all addresses. For example, the Nicaraguan address does not contain a ZIP code. And the German address does not contain a state. Also, the Nicaraguan state is fully spelled out: Managua. On the contrary, the USA state is a two letter abbreviation: MA for Massachusetts. One more thing that might not be apparent is that the USA ZIP code belongs to the state of Massachusetts. All of this is important because the module does validation of addresses. The destination is the custom ud_address content type created by the module.

Available subfields

The Address field has 13 subfields available. They can be found in the schema() method of the AddresItem class. Fields are not required to have a one-to-one mapping between their schema and the form widgets used for entering content. This is particularly true for addresses because input elements, labels, and validations change dynamically based on the selected country. The following is a reference list of all subfields for addresses:

1. langcode for language code.
2. country_code for country.
3. administrative_area for administrative area (e.g., state or province).
4. locality for locality (e.g. city).
5. dependent_locality for dependent locality (e.g. neighbourhood).
6. postal_code for postal or ZIP code.
7. sorting_code for sorting code.
8. address_line1 for address line 1.
9. address_line2 for address line 2.
10. organization for company.
11. given_name for first name.
12. additional_name for middle name.
13. family_name for last name:

Properly describing an address is not trivial. For example, there are discussions to add a third address line component. Check this issue if you need this functionality or would like to participate in the discussion.

Address subfield mappings

In the example, only 9 out of the 13 subfields will be mapped. The following code snippet shows how to do the processing of the address field:

field_ud_address/given_name: first_name
field_ud_address/family_name: last_name
field_ud_address/organization: company
field_ud_address/address_line1:
  plugin: default_value
  default_value: 'It is a secret ;)'
field_ud_address/address_line2:
  plugin: default_value
  default_value: 'Do not tell anyone :)'
field_ud_address/locality: city
field_ud_address/administrative_area: state
field_ud_address/postal_code: zip
field_ud_address/country_code: country

The mapping is relatively simple. You specify a value for each subfield. The tricky part is to know the name of the subfield and the value to store in it. The format for an address component can change among countries. The easiest way to see what components are expected for each country is to create a node for a content type that has an address field. With this example, you can go to /node/add/ud_address and try it yourself. For simplicity sake, let’s consider only 3 countries:

• For USA, city, state, and ZIP code are all required. And for state, you have a specific list form which you need to select from.
• For Germany, the company is moved above first and last name. The ZIP code label changes to Postal code and it is required. The city is also required. It is not possible to set a state.
• For Nicaragua, the Postal code is optional. The State label changes to Department. It is required and offers a predefined list to choose from. The city is also required.

Pay very close attention. The available subfields will depend on the country. Also, the form labels change per country or language settings. They do not necessarily match the subfield names. Moreover, the values that you see on the screen might not match what is stored in the database. For example, a Nicaraguan address will store the full department name like Managua. On the other hand, a USA address will only store a two-letter code for the state like MA for Massachusetts.

Something else that is not apparent even from the user interface is data validation. For example, let’s consider that you have a USA address and select Massachusetts as the state. Entering the ZIP code 55111 will produce the following error: Zip code field is not in the right format. At first glance, the format is correct, a five-digits code. The real problem is that the Address module is validating if that ZIP code is valid for the selected state. It is not valid for Massachusetts. 55111 is a ZIP code for the state of Minnesota which makes the validation fail. Unfortunately, the error message does not indicate that. Nine-digits ZIP codes are accepted as long as they belong to the state that is selected.

Note: If you are upgrading from Drupal 7, the D8 Address module offers a process plugin to upgrade from the D7 Address Field module.

Finding expected values

Values for the same subfield can vary per country. How can you find out which value to use? There are a few ways, but they all require varying levels of technical knowledge or access to resources:

• You can inspect the source code of the address field widget. When the country and state components are rendered as select input fields (dropdowns), you can have a look at the value attribute for the option that you want to select. This will contain the two-letter code for countries, the two-letter abbreviations for USA states, and the fully spelled string for Nicaraguan departments.
• You can use the Devel module. Create a node containing an address. Then use the devel tab of the node to inspect how the values are stored. It is not recommended to have the devel module in a production site. In fact, do not deploy the code even if the module is not enabled. This approach should only be used in a local development environment. Make sure no module or configuration is committed to the repo nor deployed.
• You can inspect the database. Look for the records in a table named node__field_[field_machine_name], if migrating nodes. First create some example nodes via the user interface and then query the table. You will see how Drupal stores the values in the database.

If you know a better way, please share it in the comments.

The commerceguys addressing library

With version 8 came many changes in the way Drupal is developed. Now there is an intentional effort to integrate with the greater PHP ecosystem. This involves using already existing libraries and frameworks, like Symfony. But also, making code written for Drupal available as external libraries that could be used by other projects. commerceguys\addressing is one example of a library that was made available as an external library. That being said, the Address module also makes use of it.

Explaining how the library works or where its fetches its database is beyond the scope of this article. Refer to the library documentation for more details on the topic. We are only going to point out some things that are relevant for the migration. For example, the ZIP code validation happens at the validatePostalCode() method of the AddressFormatConstraintValidator class. There is no need to know this for a migration project. But the key thing to remember is that the migration can be affected by third-party libraries outside of Drupal core or contributed modules. Another example, is the value for the state subfield. Address module expects a subdivision as listed in one of the files in the resources/subdivision directory.

Does the validation really affect the migration? We have already mentioned that the Migrate API bypasses Form API validations. And that is true for address fields as well. You can migrate a USA address with state Florida and ZIP code 55111. Both are invalid because you need to use the two-letter state code FL and use a valid ZIP code within the state. Notwithstanding, the migration will not fail in this case. In fact, if you visit the migrated node you will see that Drupal happily shows the address with the data that you entered. The problems arrives when you need to use the address. If you try to edit the node you will see that the state will not be preselected. And if you try to save the node after selecting Florida you will get the validation error for the ZIP code.

This validation issues might be hard to track because no error will be thrown by the migration. The recommendation is to migrate a sample combination of countries and address components. Then, manually check if editing a node shows the migrated data for all the subfields. Also check that the address passes Form API validations upon saving. This manual testing can save you a lot of time and money down the road. After all, if you have an ecommerce site, you do not want to be shipping your products to wrong or invalid addresses. ;-)

Technical note: The commerceguys/addressing library actually follows ISO standards. Particularly, ISO 3166 for country and state codes. It also uses CLDR and Google's address data. The dataset is stored as part of the library’s code in JSON format.

Migrating countries and zone fields

The Address module offer two more fields types: Country and Zone. Both have only one subfield value which is selected by default. For country, you store the two-letter country code. For zone, you store a serialized version of a Zone object.

What did you learn in today’s blog post? Have you migrated address before? Did you know the full list of subcomponents available? Did you know that data expectations change per country? Please share your answers in the comments. Also, I would be grateful if you shared this blog post with others.

Next: Introduction to paragraphs migrations in Drupal

This blog post series, cross-posted at UnderstandDrupal.com as well as here on Agaric.coop, is made possible thanks to these generous sponsors: Drupalize.me by Osio Labs has online tutorials about migrations, among other topics, and Agaric provides migration trainings, among other services.  Contact Understand Drupal if your organization would like to support this documentation project, whether it is the migration series or other topics.

We are Agaric, a web development collective eager to understand where you want to go with your web presence and to apply our knowledge and skills to get you there.

Because we are a worker-owned cooperative, every person you interact with at Agaric is an owner of the company.  Because we are part of the Free Software movement, we build on tools and frameworks created and tested by thousands of people.  Our goal is to give you power over your web sites and online technology.

Have something you think we may be able to help with? Just ask.

I had a great time at BioRAFT Drupal Nights, on January 16th, 2014. Originally Chris Wells (cwells) was scheduled to speak, unfortunately he was down with the flu. Michelle Lauer (Miche) put an awesome presentation together on really short notice. With help from Diliny Corlesquet (Dcor) there was plenty to absorb intellectually along with the delicious Middle Eastern food. I love spontaneity, so it was fun to hop over to BioRAFT in Cambridge, MA, to be a member of a panel of Senior Drupal Developers.

I joined Seth Cohn, Patrick Corbett, Michelle Lauer and Erik Peterson to present some viable solutions to performance issues in Drupal.
Several slides covered the topic - Drupal High Performance and a lively discussion ensued. The panel members delved into some of the top issues with Drupal performance under several different conditions. The discussion was based on the book written by Jeff Shelton, Naryan Newton, and Nathaniel Catchpole - High Performance Drupal - O'Reilly - http://shop.oreilly.com/product/0636920012269.do .

The room at BioRAFT was comfortably full with just the right amount of people to have an informal Q and A, and to directly address some concerns that developers were actually working on. Lurking in the back I spotted some of the top developers in the Drupal community (scor, mlncn, kay_v just to name a few) that were not on the panel, but they did had a lot of experience to speak from during Q and A.

The video posted at the bottom is packt with great information and techniques - Miche even shared some code snippets that will get you going if you seek custom performance enhancements.

One of the discussions was around tools that can be used to gauge performance or tweak it.

• Drupals internal syslog
• watchdog - dblog
• Acquia has some tools
• Google Analytics

I also found an online test site: http://drupalsitereview.com

We also talked a lot about caching and the several options that exist to deal with cache on a Drupal site.
There are options to cache content for anonymous users and there are solutions and modules for more robust sites with high traffic.

PHP 5.5 offers a good caching solution with integrated opcode caching. They are a performance enhancement and extension for PHP. Some sites have shown a 3x performance boost by using opcode caching. Opcode caching presents no side-effects beyond extra memory usage and should always be used in production environments. Below are a couple of the suggestions we discussed that are listed in the Modules listed on Drupal.org:

Boost

a good solution for sites that have a lot of content for anonymous users. https://drupal.org/project/boost

Varnish

a good choice for sites with content for members https://drupal.org/project/varnish

Others

• Memcache
• Redis
• File Cache

The panel agreed that there are many things to consider when seeking to improve the sites performance:

• Hosting
• Custom Code
• Cache
• Memory Limits
• Traffic
• Module Compatibility
• CSS and JS Aggregation

A few more are discussed in the video below...

Sending out Be Wells to cwells! We look forward to a future presentation, and we are also pleased that on such short notice it did not seem too hard to gather a panel of several senior Drupal developers to discuss High Performance Drupal. See the whole discussion on YouTube