We are a worker-owned cooperative specializing in building tools and websites that respect your freedom. We also provide training and consultation to meet your goals. Everything we do is part of our purpose to help all people gain the most power possible over their own lives.

Video embed

We have presented several examples as part of this migration blog post series. They started very simple and have been increasing in complexity. Until now, we have been rather optimistic. Get the sample code, install any module dependency, enable the module that defines the migration, and execute it assuming everything works on the first try. But Drupal migrations often involve a bit of trial and error. At the very least, it is an iterative process. Today we are going to talk about what happens after import and rollback operations, how to recover from a failed migration, and some tips for writing definition files.

Importing and rolling back migrations

When working on a migration project, it is common to write many migration definition files. Even if you were to have only one, it is very likely that your destination will require many field mappings. Running an import operation to get the data into Drupal is the first step. With so many moving parts, it is easy not to get the expected results on the first try. When that happens, you can run a rollback operation. This instructs the system to revert anything that was introduced when the migration was initially imported. After rolling back, you can make changes to the migration definition file and rebuild Drupal’s cache for the system to pick up your changes. Finally, you can do another import operation. Repeat this process until you get the results you expect. The following code snippet shows a basic Drupal migration workflow:

# 1) Run the migration.
$ drush migrate:import udm_subfields

# 2) Rollback migration because the expected results were not obtained.
$ drush migrate:rollback udm_subfields

# 3) Change the migration definition file.

# 4) Rebuild caches for changes to be picked up.
$ drush cache:rebuild

# 5) Run the migration again
$ drush migrate:import udm_subfields

The example above assumes you are using Drush to run the migration commands. Specifically, the commands provided by Migrate Run or Migrate Tools. You pick one or the other, but not both as the commands provided for two modules are the same. If you were to have both enabled, they will conflict with each other and fail.
Another thing to note is that the example uses Drush 9. There were major refactorings between versions 8 and 9 which included changes to the name of the commands. Finally, udm_subfields is the id of the migration to run. You can find the full code in this article.

Tip: You can use Drush command aliases to write shorter commands. Type drush [command-name] --help for a list of the available aliases.

Technical note: To pick up changes to the definition file, you need to rebuild Drupal’s caches. This is the procedure to follow when creating the YAML files using Migrate API core features and placing them under the migrations directory. It is also possible to define migrations as configuration entities using the Migrate Plus module. In those cases, the YAML files follow a different naming convention and are placed under the config/install directory. For picking up changes, in this case, you need to sync the YAML definition using configuration management workflows. This will be covered in a future entry.

Stopping and resetting migrations

Sometimes, you do not get the expected results due to an oversight in setting a value. On other occasions, fatal PHP errors can occur when running the migration. The Migrate API might not be able to recover from such errors. For example, using a non-existent PHP function with the callback plugin. Give it a try by modifying the example in this article. When these errors happen, the migration is left in a state where no import or rollback operations could be performed.

You can check the state of any migration by running the drush migrate:status command. Ideally, you want them in Idle state. When something fails during import or rollback, you would get the Importing or Rolling back states. To get the migration back to Idle, you stop the migration and reset its status. The following snippet shows how to do it:

# 1) Run the migration.
$ drush migrate:import udm_process_intro

# 2) Some non recoverable error occurs. Check the status of the migration.
$ drush migrate:status udm_process_intro

# 3) Stop the migration.
$ drush migrate:stop udm_process_intro

# 4) Reset the status to idle.
$ drush migrate:reset-status udm_process_intro

# 5) Rebuild caches for changes to be picked up.
$ drush cache:rebuild

# 6) Rollback migration because the expexted results were not obtained.
$ drush migrate:rollback udm_process_intro

# 7) Change the migration definition file.

# 8) Rebuild caches for changes to be picked up.
$ drush cache:rebuild

# 9) Run the migration again.
$ drush migrate:import udm_process_intro

Tip: The errors thrown by the Migrate API might not provide enough information to determine what went wrong. An excellent way to familiarize yourselves with the possible errors is by intentionally breaking working migrations. In the example repository of this series, there are many migrations you can modify. Try anything that comes to mind: not leaving a space after a colon (:) in a key-value assignment; not using proper indentation; using wrong subfield names; using invalid values in property assignments; etc. You might be surprised by how Migrate API deals with such errors. Also, note that many other Drupal APIs are involved. For example, you might get a YAML file parse error, or an Entity API save error. When you have seen an error before, it is usually faster to identify the cause and fix it in the future.

What happens when you rollback a Drupal migration?

In an ideal scenario, when a migration is rolled back, it cleans after itself. That means, it removes any entity that was created during the import operation: nodes, taxonomy terms, files, etc. Unfortunately, that is not always the case. It is very important to understand this when planning and executing migrations. For example, you might not want to leave taxonomy terms or files that are no longer in use. Whether any dependent entity is removed or not has to do with how plugins or entities work.

For example, when using the file_import or image_import plugins provided by Migrate File, the created files and images are not removed from the system upon rollback. When using the entity_generate plugin from Migrate Plus, the created entity also remains in the system after a rollback operation.

In the next blog post, we are going to start talking about migration dependencies. What happens with dependent migrations (e.g., files and paragraphs) when the migration for host entity (e.g., node) is rolled back? In this case, the Migrate API will perform an entity delete operation on the node. When this happens, referenced files are kept in the system, but paragraphs are automatically deleted. For the curious, this behavior for paragraphs is actually determined by its module dependency: Entity Reference Revisions. We will talk more about paragraphs migrations in future blog posts.

The moral of the story is that the behavior migration system might be affected by other Drupal APIs. And in the case of rollback operations, make sure to read the documentation or test manually to find out when migrations clean after themselves and when they do not.

Note: The focus of this section was content entity migrations. The general idea can be applied to configuration entities or any custom target of the ETL process.

Re-import or update migrations

We just mentioned that Migrate API issues an entity delete action when rolling back a migration. This has another important side effect. Entity IDs (nid, uid, tid, fid, etc.) are going to change every time you rollback an import again. Depending on auto generated IDs is generally not a good idea. But keep it in mind in case your workflow might be affected. For example, if you are running migrations in a content staging environment, references to the migrated entities can break if their IDs change. Also, if you were to manually update the migrated entities to clean up edge cases, those changes would be lost if you rollback and import again. Finally, keep in mind test data might remain in the system, as described in the previous section, which could find its way to production environments.

An alternative to rolling back a migration is to not execute this operation at all. Instead, you run an import operation again using the update flag. This tells the system that in addition to migrating unprocessed items from the source, you also want to update items that were previously imported using their current values. To do this, the Migrate API relies on source identifiers and map tables. You might want to consider this option when your source changes overtime, when you have a large number of records to import, or when you want to execute the same migration many times on a schedule.

Note: On import operations, the Migrate API issues an entity save action.

Tips for writing Drupal migrations

When working on migration projects, you might end up with many migration definition files. They can set dependencies on each other. Each file might contain a significant number of field mappings. There are many things you can do to make Drupal migrations more straightforward. For example, practicing with different migration scenarios and studying working examples. As a reference to help you in the process of migrating into Drupal, consider these tips:

• Start from an existing migration. Look for an example online that does something close to what you need and modify it to your requirements.
• Pay close attention to the syntax of the YAML file. An extraneous space or wrong indentation level can break the whole migration.
• Read the documentation to know which source, process, and destination plugins are available. One might exist already that does exactly what you need.
• Make sure to read the documentation for the specific plugins you are using. Many times a plugin offer optional configurations. Understand the tools at your disposal and find creative ways to combine them.
• Look for contributed modules that might offer more plugins or upgrade paths from previous versions of Drupal. The Migrate ecosystem is vibrant, and lots of people are contributing to it.
• When writing the migration pipeline, map one field at a time. Problems are easier to isolate if there is only one thing that could break at a time.
• When mapping a field, work on one subfield at a time if possible. Some field types like images and addresses offer many subfields. Again, try to isolate errors by introducing individual changes each time.
• There is no need to do every data transformation using the Migrate API. When there are edge cases, you can manually update those after the automated migration is completed. That is, no more rollback operations. You can also clean up the source data in advance to make it easier to process in Drupal.
• Commit to your code repository any and every change that produces the right results. That way, you can go back in time and recover a partially working migration.
• Learn about debugging migrations. We will talk about this topic in a future blog post.
• Seek help from the community. Migrate maintainers and enthusiasts are very active and responsive in the #migrate channel of Drupal slack.
• If you feel stuck, take a break from the computer and come back to it later. Resting can do wonders in finding solutions to hard problems.

What did you learn in today’s blog post? Did you know what happens upon importing and rolling back a migration? Did you know that in some cases, data might remain in the system even after rollback operations? Do you have a use case for running migrations with the update flag? Do you have any other advice on writing migrations? Please share your answers in the comments. Also, I would be grateful if you shared this blog post with your colleagues.

Next: Migrating files and images into Drupal

This blog post series, cross-posted at UnderstandDrupal.com as well as here on Agaric.coop, is made possible thanks to these generous sponsors. Contact Understand Drupal if your organization would like to support this documentation project, whether it is the migration series or other topics.

The ever intensifying climate crisis is an existential threat. It was brought upon by people designing and building technology reliant upon the extraction and burning of fossil fuel. That and an economic system reliant on ever increasing consumption, demanding energy production unsustainable to the planet.

From September 20th through September 27th millions of people across the globe skipped school, walked out of workplaces, and joined in the streets to demand bold, swift action for climate justice. Many in the tech industry, including ourselves, went on digital strike, shuttering our websites for the day to join in the action. In fact, we partnered with 350.org to improve the mapping tool used to help people organize and find climate strike actions.

But what does bold, swift action mean? What is required of us to respond to the enormity of this climate catastrophe?

One thing is for certain - it won't be fixed with the same values, systems and forces that ushered in this emergency.

There are many ideas out there on what we should do, with many different names. Whatever the specifics, the path to climate justice rests on our ability to move away from extractive relationships with our earth and quickly grow regenerative relationships instead. Many are calling this the Commons Transition. Those of us working in the tech industry have tremendous influence in participating in and assisting with this transition.

What is the Commons?

The Commons is a way to organize and manage resources collaboratively among the community of producers and users. They exist outside of both the public and private sector. The commons is flourishing all around us, particularly where indigenous communities have been able to defend themselves and their land. The Zapatista caracoles in Chiapas, Mexico, the confederalist communes of Rojava, and the Potato Park in Peru are just a few examples. All are a blend of retaining generations-long wisdom of living in right relation, an unlearning of the extractive patterns that have gained dominance, and innovating new ways of commoning.

Free and open-source software movements put into practice much of the Commons' values. Combining this approach to software development with cooperative, community-based economic models can help us build technology that is sustainable and helps facilitate and defend the commons.  

The Tech Industry's Role in the Commons Transition

As mentioned, the commons is all around us. Yet these commons are constantly battling enclosure, and many more commons projects need our support to take hold. The first step is to take stock of the commons we are currently part of. Millions of us are already members of credit unions, food cooperatives, worker-owners in cooperatives, purchasers of cooperatively produced goods and visitors to cooperatively managed parks and open spaces.

This morning, I walked to the Westwood Food Cooperative to buy produce and am now typing this essay at Kahlo's, a Mexican-American family-run restaurant with a plant-heavy menu. I am a worker-owner of Agaric, a tech cooperative. We're also members of MayFirst, a cooperative itself which provides the hosting infrastructure for our clients. The commons are everywhere.

How might we better support the commons we are already part of?

The best way to find is to ask. To find commons near you, visit SolidarityEconomy.us. It's not comprehensive, but it's a good start.

Once better connected to the commons you are part of, examine what other aspects of your life can align with the commons. If you tend to shop at a big box store, look into local food co-ops and community supported agriculture (CSA). If you work for an agency, consider how you might transition it to a cooperative with a social-mission prioritizing sustainable practices. If you don't see your company moving in a democratic direction, you can still shift it that way by unionizing. These are tall orders, but then again, these times call for tall orders.

Lead By Example, But Towards Systemic Change

The quickest way to effect change is to work within our spheres of influence. But one person shopping at a food co-op instead of Wal-Mart is just a drop in the bucket. How do we make our individual actions add up to the systemic change we need? In the essay "We Can't Do It Ourselves,"
published in Low Tech Magazine, Kris De Decker explains,

A sustainability policy that focuses on systemic issues reframes the question from “how do we change individuals’ behaviours so that they are more sustainable?” to “how do we change the way society works?”.  

De Decker elaborates that,

Addressing the sociotechnical underpinnings of “behaviour” involves attempting to create new infrastructures and institutions that facilitate sustainable lifestyles, attempting to shift cultural conventions that underpin different activities, and attempting to encourage new competences that are required to perform new ways of doing things.  

Low Tech Magazine is leading by example by running their website completely off of solar power they themselves produce. I encourage fellow designers and developers to check out the site and the detailed write up on the meticulous, thorough work they did to minimize the site's energy usage and rig up solar panels to power the server the site runs on.

Similarly, however we can, as tech workers we should be building our technology sustainably and in service of regenerative relations.

Use and Demand Renewable Energy

We must move away from fossil fuels and towards clean, renewable energy. For Agaric, this means getting the hosting providers we partner with to use clean, renewable energy. Most of our sites run on MayFirst, a tech cooperative we are members and leaders within. I started a discussion thread to transition our tech stack to clean, renewable sources . The discussion has become an initiative. Our work is just getting started, but this shows the benefit of meeting our needs democratically.   

You can see where your hosting providers stand by using https://ecograder.com

If your host isn't on 100% renewables, talk to them and work to get them there.

You can also choose to go with a hosting provider that's already green, such as https://greenhost.net

Design for Energy Efficiency

Web development trends have lead to massive growth in the size and resource usage of websites. There are counter trends appearing in response - static site generators, lean content strategies and sustainable design. Just as we care about the energy efficiency of our cars, homes and appliances we should care too about the carbon footprint of our websites and apps. Plus, simpler, efficient online tools facilitate easier maintenance and faster load times for users (especially important for those on limited data plans and older devices). As contributors and proponents of Drupal, there is certainly room for improvement within our practice for more sustainable design.

Divest, Invest for the Transition

Technology is not neutral. What it is put into service of has tremendous impact. The climate strike demands of the Tech Workers Coalition are a great starting point.

•  Zero carbon emissions by 2030
•  Zero contracts with fossil fuel companies
•  Zero funding of climate denial lobbying or other efforts
•  Zero harm to climate refugees and frontline communities

Work to formalize these principles at your own workplace and make them public. If you have contracts with fossil fuel companies or climate deniers work to end them. A great group to link up with around this is ClimateAction.tech

Unionize to Build Counter Power

Not all of us work in democratic workplaces. For most of us, creating change in hierarchical startups and corporations is more difficult. Most companies are oriented towards shareholder needs, not worker or planet needs. Unions build counter power, forcing business to take our needs into account.

This is a daunting task. However, worker self-organizing is on the rise, especially in our industry. What starts as an internal petition signed and sent to management, can build to lasting formations that can shift companies to align with the values we need to make the transition.

Joining the Campaign to Organize Digital Employees is a great way to get started.

Conclusion

Transitions aren't easy. They're uncomfortable, they require risk and there's no guarantee of success. Still, fixing our sights on a world of care, balance and right relation with the earth and then making it so, in our home and in our workplace is deeply rewarding.

Industry audit and competitive review

You may in many ways be without peer, but there are always competitors for the attention of your audience. Identifying top peers and reviewing their respective content helps you get a wider perspective both on what potential listeners, members, and donors will be seeing and what seems to be working for others— we can start thinking together about where to emulate and where to differentiate, informing all of our work together.

Content strategy

Building on the review of peers, Agaric will work with you to briefly interview current and potential clients and develop personas and user stories.

Content style guide

Along with bringing consistency to cooperative output (and saving time sweating the details every time they come up), a good content (copywriting) style guide incorporates suggestions for clear and effective writing and helps your unique aspects shine through. It can help tell your story in a consistent way and help let your individual personalities show through while maintaining collective coherence.

User research and testing

Agaric applies a Lean UX Research methodology to answer critical user experience questions with relevant, meaningful, and actionable data.

From reviewing your goals and audiences we recommend answering the following questions:

• What prevents users from becoming paying subscribers or sustaining donor members on the site?
• What features would enable users to invite friends to become supporters?
• Who is most engaged? least engaged?
• Which conversion goals are being met? unmet?
• What new audience do we want to reach?

We recommend and use the following research and testing approaches:

• A/B testing
• Brand audit
• Browser and device analysis
• Competitive analysis
• Content audit
• Contextual inquiry
• Heat map analysis
• Heuristic evaluation
• Site analytics review
• SWOT (Strengths, Weaknesses, Opportunities, Threats) analysis
• Usability testing
• User interviews
• User surveys

Not all will fit the purpose or budget of every part of a project, but good insights into what to build and why is more valuable than simply building well and quickly.

User testing & revisions

We always recommend at least one round dedicated to measuring and improving. Using analytics and user tests, we identify what is working, what is not and needs to be changed, and what is missing and needs to be built. We then build on the previous work to do the fixes and enhancements with the highest expected impact.

What is Free Software?

A program is free software if the program's users have the four essential freedoms:

1. The freedom to run the program as you wish, for any purpose (freedom 0).
2. The freedom to study how the program works, and change it, so it does your computing as you wish (freedom 1).
3. The freedom to redistribute copies so you can help others (freedom 2).
4. The freedom to distribute copies of your modified versions to others (freedom 3).

Access to the source code is a precondition for this. To determine whether the source code is free, see what license it carries and then check the GNU project's list of licenses.

We do a few tasks that we do not have free software for. So, we use non-free programs for them. Sometimes we use specific non-free software that a client insists on including in the web site. By using non-free programs, we sacrifice some of our freedom; some activists would refuse to do that. We do not like the compromise, so we help develop free replacements for those programs. Whenever possible, we use software that reflects our values.

Operating Systems

GNU/Linux

We have chosen to use GNU/Linux as our default system for our local development. When we take on a new student, we install a GNU/Linux distribution. We always give the option of installing a different distribution, or if a student wishes to do so, they may. These are the favored GNU/Linux distributions in use by our cooperative team *members:

• Debian
• Ubuntu
• Pop! OS (an Ubuntu derivative)

These are not the best versions of GNU/Linux in regards to being completely free; you should consult the list of free distributions on the Free Software Foundation website.

* Currently one team member is using the proprietary but BSD-based Mac OS X, which is compliant with the Unix 03 / POSIX standard which GNU/Linux distributions also meet.

Browsers

Firefox

As developers, we have to test client sites in all browsers, but for working and building sites, we use Mozilla Firefox. Although the source code of Firefox is free software, they include some icons, trademarks, and logos in the download that make it non-free. You can easily remove these as has been done with IceCat, the GNU version of the Firefox browser. It has great performance, developer tools, and community. The plus side of having a community around the software we use is having access to a large pool of people with experience and guidance as we learn and build together.

Tor Browser

As citizens, we are not fond of being tracked, so we use a free anonymizing web browser that does not allow tracking. It is called Tor.

Micky uses and loves the Brave browser on Android and her GNU/Linux laptop. It blocks ADs right out of the box!  https://brave.com/

Without looking deeply at Qwant, it looks pretty decent - https://www.qwant.com/

We came across Qwant via https://iridiumbrowser.de/ - a more secure chromium release, but probably has some things we may not know about or want...

File Storage and Calendar

NextCloud

NextCloud is a collection of utilities suited for the De-Googlization process. It provides most of the popular tools in Google Drive. Agaric uses a hosted version of NextCloud on MayFirst.org servers that is inclusive of:

• document and file storage
• shared document editing
• multi-device synchronization
• image galleries
• calendar
• contacts

See a comparison of the features NextCloud offers vs. the proprietary options like GoogleDocs/Drive.

Finance, Accounting, and Bookkeeping

GNUcash

Accounting software that we use for our bookkeeping.

You can see a review of GNUcash vs. Quickbooks and decide if it works for you. We have found a few bookkeeping cooperatives that do accounting with GnuCash.

Real-time Chat

As a team, we rely on different tools to communicate with each other and with clients about daily activities and long term project goals. We have a distributed team at locations around the world and must maintain contact especially when pair-programming or during a migration which calls for all-hands-on-deck, as well as sharing some long informational text notes and documents that include administrative information.

Freenode

IRC - Internet Relay Chat - Realtime Text Chat: Yes, we still use IRC, and you can find us on irc.freenode.net server in the #devs-r-us channel

Our preferences here are as varied as our team members: some use irssi via a remote, always-on virtual server, many use desktop clients, such as HexChat or Konversation, and still, others prefer the web-based solution "The Lounge."

Email

MayFirst.org hosts Agaric.com email

Email Client: Thunderbird

An email client from Mozilla, which also makes Firefox, and is available for your phone. It also has an encryption plugin called EnigMail that works well and is not super tricky to get set up.

Hosted Email: RiseUp: Encrypted services run by anonymous volunteers and you must be invited to have a membership.

MayFirst offers three web-based email solutions.

1. Roundcube which has a friendly and simple web interface, making it the easier of the two programs to use.
2. SquirrelMail is an option that is Javascript-free!
3. Horde, on the other hand, offers more than just email - you can share calendars, to-dos and more with other members of your group.

Hosted Email

Protonmail: An email service that is hosted and encrypted.

Email Lists:

We use email list servers for mailing lists based on groups and topics. It allows group mailing to people that sign up for a specific list.

MayFirst Email Server

RiseUp Email Server

Social Media

Mastodon: Publish anything you want: links, pictures, text, video. All on a platform that is community-owned and ad-free.

Social.coop: A community similar to Twitter, the main difference is that it is owned by the members. For as little as $1 a month you can become an owner/member and take part in shaping the future of the platform. You can find and follow Agaric in social.coop, a coop-run corner of the fediverse, a cooperative and transparent approach to operating a social platform

Live Streaming

MayFirst Live Streaming: MayFirst membership includes live streaming.

Conference Calls and Online Meetings

Some Agaric team members are using Jitsi recognizing that it is a work in progress and there may be technical failures at times - such as we have also found using Google Hangouts - lag time, cut-offs, poor sound quality and issues with screen sharing. At times we have found that we need to use a proprietary solution that seems to work reliably as we continue to support development efforts and bug fixes with Jitsi. At the heart of Jitsi are Jitsi Videobridge and Jitsi Meet, which let you have conferences on the internet, while other projects in the community enable other features such as audio, dial-in, recording, and simulcasting.

You can self-host an instance of Jitsi or choose a hosted version. You can use http://meet.jit.si or an instance is also available for public use at https://meet.mayfirst.org We do encourage you to become a MayFirst member and have access to all of the free software tools they offer. The Jitsi project needs volunteers to use and test Jitsi so it can get better swiftly!

Currently, Agaric was using and paying for, the proprietary Zoom audio/video conference call service and software. Since October we have been using BigBlueButton a free software alternative. BBB has been working well for us and we have worked it into being our go-to solution for protecting our privacy.

BigBlueButton

BigBlueButton.org: Agaric uses BigBlueButton as our video-chat meeting platform. We also offer free chatroom hosting for those that cannot afford to pay for this service. 

Phone Calls and Text Messages

Signal: Agaric uses Signal to encrypt SMS text messages and phone calls. Encrypted phone and instant messaging found to be secure and recommended by Edward Snowden as the only truly encrypted messaging app that is not able to be decrypted by anyone. Note that security is an arms race and this could become false at any time.

Collaborative Note Taking

Etherpad: When hosting an online meeting we usually open a shared note pad so that everyone may contribute to getting the important bits logged. Etherpad text is synchronized as you type so that everyone viewing the page sees the same text. This allows you to collaborate on documents with large or small teams seamlessly! We use the hosted version, but you are welcome to host it yourself. We have tried a few online pads and settled on Etherpad as the most reliable.

Collaborative Ongoing Discussion

With some collaborators, particularly people involved with the Drutopia initiative, we use Mattermost rather than IRC. Mattermost can be more useful for ongoing discussions; it is similar to Slack and offers a threaded conversation. The community version is free software.

Notes and ToDo Lists

TomBoy A tiny app that lets you take notes while it conveniently makes hyperlinks out of titles and allows synchronization over SSH and more.

Password Management

KeePass A password management system that takes most of the worry, distraction and thinking out of storing and retrieving your login information for multiple projects and sites.

Text Document Editing, Spreadsheets, and Presentations

Libre Office: A suite of office tools similar to Microsoft Office, Documents, Spreadsheets, Slides. We use LibreOffice tools that come as core software in the distributions of GNU/Linux we are using. You may have heard of OpenOffice; it is now called LibreOffice. It consists of basic publishing and calculating software for doing office tasks. These are the ones we use most often:

1. LibreOffice Calc - Similar features and functions of a calculating software to make spreadsheets, such as MicroSoft Excel

2. LibreOffice Writer - Similar features and functions of a word processor such as MicroSoft Word

3. LibreOffice Impress - We use this tool to build slide decks and presentations using text/graphics and videos; it is similar to Microsoft PowerPoint in features.

Project Management and Issue Tracking

*GitLab: This tool is a web-based and self-hosted Git-repository manager with a wiki and issue-tracking features. We also use Gitlab for cooperative development on our projects.

*Although GitLab isn't fully free software, it does offer a self-hosted version that is. The Enterprise hosted version has extra features and is proprietary.

Redmine: A free program that you can run locally or on your own server for use as a project management and issue tracking tool. Before finding GitLab, we used a self-hosted instance of Redmine which is free software.

Decision Making and Voting

Loomio: A hosted service available at http://loomio.org

Loomio offers distributed decision-making system where you can make groups that can have discussions and make decisions without an in-person meeting. Decide yes or no, or that you need more information.

Note that Loomio also has built a great cooperative resource on at their other URL - http://loomio.coop

Customer Relationship Management

CiviCRM: Agaric is working with the developers at MyDropWizard to take a look at CiviCRM with Drupal 8.

CiviCRM is a free software to manage client relationships and memberships. We have not deployed it yet.

Resources and Free Software Directories

• Free Software Directory
• Awesome-list of self-hosted applications
• You can search for Free software alternatives. - be sure to search for Free/Open Source, not the free trial software, and LOOK at the license.

You can contribute to groups working towards solutions, there are many roles, and you do not have to be a developer. As an example, *IndieWeb and Jitsi are projects that we make time to support with development, testing, outreach, and feedback.

*With IndieWeb, you can take control of your articles and status messages can go to all services, not just one, allowing you to engage with everyone. Even replies and likes on other services can come back to your site, so they’re all in one place.

Framasoft: A large collection of free software tools where we use the calendar and polling software most often. We are experimenting with several other FramaSoft tools and may adopt them in the future.

If this has been a helpful read, please pass it on and let us know in the comments how it helped you. A follow-up post will list the tools we use for development purposes. Please be sure to mention any free software you have found and are using now.

I will leave you with an excellent TEDx talk where Richard Stallman explains Free Software:

Sign up to be notified when Agaric gives a migration training:

Image credit: "Anti-Muslim Ban Protests" by Mrs. Gemstone is licensed under CC BY-SA 2.0.

Stay in touch

Drutopia Features

Actions

An action is a call for concrete work from supporters. It can stand on its own or be part of a campaign. It can have a goal and due date set. For example, a call for 50 people to call a prison by June 13th demanding an investigation into prisoner abuse. An action can also be turned into a fundraiser, allowing people to donate directly to the site through Stripe. No longer will groups be dependent on proprietary platforms such as GoFundMe and IndieGoGo.

Articles

Drutopia provides an article content type and related configuration, including article types (such as "Victories" or "Press releases") and topics which span all kinds of content on the site (such as "Tenant organizing" or "Environmental racism").

Blog Posts

Drutopia provides the blog content type and related configuration, ideal for giving individuals or groups more informal voices distinct from the organization as a whole.

Campaigns

A campaign page is a central place to explain an issue, publish updates about its activity, list out demands, post calls to action, and raise funds.

Groups

• Groups broadcast to supporters their activity and ways to support.
• Each group has a page within the website. A group can launch a campaign, call for an action and publish a news article.
• All of these pieces of content are displayed on a group's page, creating a mini-site.

Profile Pages

Drutopia provides a People content type for showing visitors information about staff, volunteers, contributors, and more.

The Drutopia Initiative

The Drutopia initiative lays out a technique for configuring a Drupal distribution, for the commons.  It is a recognition and a reflection in software architecture that liberation requires organizing and collective action. We ask you not to sit back and let Drutopia work its magic, but rather to stand up and, with us, build a cooperative of grassroots organizations that work together to build tools that we can own together.

Core values of the Drutopia initiative:

• Be inclusive regarding gender, gender identity, sexual orientation, ethnicity, ability, age, religion, geography, and class.
• Commit to protection of personal information and privacy and freedom from surveillance
• Value collaboration and cooperation above competition
• Prioritize human needs over private profit
• Foster non-hierarchical structures and collective decision-making

Drutopia encourages:

• Users and adopters to pool their resources and take a lead role in prioritizing and crowdfunding new development
• Designers, site builders, and developers to focus on rolling their work into shared solutions available to everyone, heightening the ability of groups of all stripes and sizes to benefit.

Strategy - Shared Configuration Management

Drutopia focuses on a specific use case for websites — grass-roots organizing — and pioneers an elegant model for reducing the costs of maintaining an individual Drupal site by providing a shared base configuration for a collection of websites that are run by various organizations with similar needs that contribute to the development cooperatively.

Learn more specifics about the cooperative development strategy behind Drutopia by reading the Drutopia White Paper.

Drutopia as a LibreSaaS or Platform Cooperative

We believe that the platforms that influence and affect so much of our lives (think Facebook and Google) should be owned by the people who use them. In the same line of thinking, the potential of the Drutopia configuration framework is truly unleashed as fully hosted, member-owned platforms. Members of the platform cooperatives will drive forward the vision of the project from the perspective of those most affected– shaping the development of new features. The platforms themselves can span multiple participating hosts that endorse and follow the hosted Drutopia standards.

Contribute to the Drutopia Initiative

Drutopia is a fully functional content management system apart from all these goals, and there are a few ways different individuals and organizations can contribute to this initiative:

1) Drutopia as a Drupal distribution that can be self-hosted by anyone, but the software project will be democratically governed by members of the Drutopia cooperative. For as little as $10 a year you can become a member who votes in leadership team elections, suggests features for the development roadmap, and is part of a community that is building tools for the grassroots.

2) Non-profits and other organizations that pay Chocolate Lilly or Agaric to build a Drutopia website for their organizing will be contributing to our development efforts, and helping us to move forward in attaining the goals of the Drutopia initiative.

3) Individuals and groups that recognize needs for specialized Drupal distributions should talk to us. We encourage you to reach out and bring together others that share the same needs, so that we can collectivize the development and maintenance of more Drutopia distributions. 

Cooperative Webhosting

Staying true to the core values of Drutopia, we hosting your site through, and include membership in, democratically governed May First Movement Technology cooperative.  May First is dedicated to supporting organizers and activists by providing tools and services that protect their data and privacy from governments and corporations.

America was founded on freedom and liberty. Eventually we found out that did not mean freedom for me, and probably not for you either. We have also discovered that when corporations sell proprietary software it prevents your access to features and compromises your privacy. So, continuing to build the future on a proprietary foundation, without total access to the inner workings {source code}... we can be held hostage at anytime to a myriad of conditions that we have no control over.

Building on proprietary solutions also means you will be excluding many of the people that need to benefit from it the most, whether intentionally or not. The people that need freedom are mostly hindered by obstacles that we can remove. Most are hindered by the lack of education about computer use and lack of money to buy the newest devices that are necessary to experience the internet. Out of 7.125 billion, 1.9 billion people have smartphones. This consumer revolution is prompting families existing on low wages to make hard choices about their lifestyles and great sacrifices to purchase an internet device such as a smartphone. How will they pay for the connectivity? Should we all be working on a free solution to this global issue?
The values identified by the Free Software Movement exemplify the values needed in a society that offers justice to all, society needs to offer access to justice for all too.

A program is free software if the program's users have the four essential freedoms:

• Freedom 0: The freedom to run the program as you wish, for any purpose.
• Freedom 1: The freedom to study how the program works, and change it so it does your computing as you wish. Access to the source code is a precondition for this.
• Freedom 2: The freedom to redistribute copies so you can help your neighbor.
• Freedom 3: The freedom to distribute copies of your modified versions to others. By doing this you can give the whole community a chance to benefit from your changes. Access to the source code is a precondition for this.

You can relate this to a person by changing a few of the words:

A person is free if they have the four essential freedoms:

• Freedom 0: The freedom to live as you wish, for any purpose.
• Freedom 1: The freedom to study what and how you wish. Access to education is a precondition for this.
• Freedom 2: The freedom to redistribute studies or ideas so you can help your neighbor.
• Freedom 3: The freedom to distribute copies of modified studies or ideas to others. By doing this you can give your whole community a chance to benefit from your experiences. Access to the education to formulate and understand ideas is a precondition for this.

So, how does this affect my daily life? After several years in the Drupal community, the free scholar in me got really tired of hearing developers that are excited about some proprietary solutions that happen to work with Drupal. Some of these solutions are being touted as the way to go in the future. I believe that our community has no shortage of genius, creative minds, and brilliant ideas. I encourage us all to think deeply about the tools we use and how they impact our personal liberties and how we can create comfortable jobs for ourselves.

For those of us that are developers, learning to explain the philosophy of free software as opposed to open source is key to autonomy, privacy, and human rights. Framing the conversation is key to the message getting to the people that need to hear it. When people are choosing convenience over privacy, it is hard to frame an argument for free software that asks us to be involved by contributing our ideas. To me, it seemed necessary to involve the people with the best future vision, that have already expounded on these theoretical scenarios around privacy concerns that have become a reality.

In May of 2015, I invited Richard M. Stallman (RMS) to the monthly Drupal meetup at MIT. I wanted him to meet some of the developers from the Boston Drupal community and experience how we work together on sharing knowledge and working on projects that are dear to our hearts, helpful to our communities, and good for society and cooperation. He gave a short lightning talk about free software and hardware followed by a Q and A. Many of us got a better understanding of what the free software movement is and how we can be a part of the educational outreach for the Free Software Foundation (FSF) to raise awareness and adoption. You can find most of the educational materials you will need on the FSF.org website.

Next, I invited RMS to NYCcamp. On Saturday July 18th, he gave the keynote speech to a large group, mostly web developers, at the United Nations. Later, he connected with the Aegir team to discuss web hosting platforms and free server tools, which led to a larger group convening at the Aegir Strategy Session with the Aegir team lead, Chris Gervais and RMS. They led an engaging round table discussion with about 30 people actively talking about the future of free tools for hosting and managing websites.

Eighteen people joined RMS for dinner and informal conversation in NYC at Congee Village. I love Drupal almost as much as I love food and freedom.

These group experiences had an affect on the community and I have since heard from several developers on how much they really enjoyed meeting RMS and knowing more about the ways free software affects our lives and our communities. One person commented "I thought RMS would be a jerk, like Linus, but he is really humble and kind when you talk to him in person."

A lot of people ask me if I work for the Free Software Foundation and my response is - no, I am a worker-owner with Agaric and we do web development using free software, but on the issues of freedom, I really work for you and me. Together, we are the mothers and fathers of our own freedoms, or not. The choice is ours to make.

What an excellent time! Your voice and thoughts are needed. Our society is only as free as the tenets on which it is built upon.
Let's free our future with free software as the foundation.

Causa jus luctus persto. Accumsan molior te vicis. Accumsan bene enim laoreet loquor tego. Defui iusto virtus. Ad adipiscing conventio fere iustum oppeto quidem validus venio. Blandit capto cui hos illum letalis torqueo. Adipiscing commodo incassum interdico lenis premo sit tincidunt. Acsi aliquip capto defui gemino genitus haero humo. Conventio diam fere velit wisi. Conventio ea facilisi hendrerit jus natu pertineo plaga quadrum vicis. Abbas amet appellatio inhibeo macto magna. Dignissim facilisi fere lucidus natu quae qui roto utinam. Esse lobortis nobis paratus premo tego valetudo vero. Lucidus olim pagus sit. Importunus luctus quibus. Aliquam melior olim velit. Abluo decet esse incassum iusto sino utinam velit. Dolore dolus nulla sagaciter scisco tation tego turpis. Euismod gilvus ideo magna proprius. Camur dolore erat incassum iusto loquor nunc. Appellatio ideo ut. Decet iustum laoreet sagaciter typicus vulputate.

Gemino quadrum vel virtus. Commodo minim vicis vulpes. Loquor nibh patria praesent torqueo usitas velit vulpes. Decet iusto mauris quia secundum suscipit vel velit vindico. Et gemino olim singularis verto volutpat. Acsi importunus nutus pala paulatim quidem. Abico accumsan caecus gemino melior paulatim valde. Abdo iaceo jumentum jus ludus nutus pagus refoveo sed vulputate. Imputo iustum secundum. Dolor eu facilisi iaceo natu secundum sino usitas. Aliquam aliquip illum neo praesent proprius. Capto erat nostrud occuro valetudo ymo. Abigo melior occuro proprius quadrum ratis similis typicus.

Appellatio augue brevitas ideo nostrud premo proprius turpis uxor. Accumsan capto enim imputo mauris pertineo quia scisco sed sino. Brevitas importunus jus minim paratus quibus refero valetudo vulpes. Adipiscing autem modo suscipere vicis. Erat haero mos probo sit. Causa elit gilvus quis suscipit ut. Caecus commodo commoveo duis fere illum importunus pneum singularis vulpes. Euismod genitus interdico probo proprius. Aptent cogo loquor metuo suscipit vulputate. Consequat exerci jumentum oppeto scisco te tincidunt typicus validus. Esca esse hendrerit jugis lobortis metuo occuro ulciscor.

Today we will learn how to migrate content from a XML file into Drupal using the Migrate Plus module. We will show how to configure the migration to read files from the local file system and remote locations. We will also talk about the difference between two data parsers provided the module. The example includes node, images, and paragraphs migrations. Let’s get started.

Note: Migrate Plus has many more features. For example, it contains source plugins to import from JSON files and SOAP endpoints. It provides many useful process plugins for DOM manipulation, string replacement, transliteration, etc. The module also lets you define migration plugins as configurations and create groups to share settings. It offers a custom event to modify the source data before processing begins. In today’s blog post, we are focusing on importing XML files. Other features will be covered in future entries.

Getting the code

You can get the full code example at https://github.com/dinarcon/ud_migrations The module to enable is UD XML source migration whose machine name is ud_migrations_xml_source. It comes with four migrations: udm_xml_source_paragraph, udm_xml_source_image, udm_xml_source_node_local, and udm_xml_source_node_remote.

You can get the Migrate Plus module using composer: composer require 'drupal/migrate_plus:^5.0'. This will install the 8.x-5.x branch where new development will happen. This branch was created to introduce breaking changes in preparation for Drupal 9. As of this writing, the 8.x-4.x branch has feature parity with the newer branch. If your Drupal site is not composer-based, you can download the module manually.

Understanding the example set up

This migration will reuse the same configuration from the introduction to paragraph migrations example. Refer to that article for details on the configuration: the destinations will be the same content type, paragraph type, and fields. The source will be changed in today's example, as we use it to explain XML migrations. The end result will again be nodes containing an image and a paragraph with information about someone’s favorite book. The major difference is that we are going to read from XML. In fact, three of the migrations will read from the same file. The following snippet shows a reduced version of the file to get a sense of its structure:

<?xml version="1.0" encoding="UTF-8" ?>
<data>
<udm_people>
<unique_id>1</unique_id>
<name>Michele Metts</name>
<photo_file>P01</photo_file>
<book_ref>B10</book_ref>
</udm_people>
<udm_people>
...
</udm_people>
<udm_people>
...
</udm_people>
<udm_book_paragraph>
<book_id>B10</book_id>
<book_details>
<title>The definite guide to Drupal 7</title>
<author>Benjamin Melançon et al.</author>
</book_details>
</udm_book_paragraph>
<udm_book_paragraph>
...
</udm_book_paragraph>
<udm_book_paragraph>
...
</udm_book_paragraph>
<udm_photos>
<photo_id>P01</photo_id>
<photo_url>https://agaric.coop/sites/default/files/pictures/picture-15-1421176712.jpg</photo_url>
<photo_dimensions>
<width>240</width>
<height>351</height>
</photo_dimensions>
</udm_photos>
<udm_photos>
...
</udm_photos>
<udm_photos>
...
</udm_photos>

Note: You can literally swap migration sources without changing any other part of the migration.  This is a powerful feature of ETL frameworks like Drupal’s Migrate API. Although possible, the example includes slight changes to demonstrate various plugin configuration options. Also, some machine names had to be changed to avoid conflicts with other examples in the demo repository.

Migrating nodes from a XML file

In any migration project, understanding the source is very important. For XML migrations, there are two major considerations. First, where in the XML tree hierarchy lies the data that you want to import. It can be at the root of the file or several levels deep in the hierarchy. You use an XPath expression to select a set of nodes from the XML document. In this article, the term element when referring to an XML document node to distinguish it from a Drupal node.  Second, when you get to the set of elements that you want to import, what child elements are going to be made available to the migration. It is possible that each element contains more data than needed. In XML imports, you have to manually include the child elements that will be required for the migration. The following code snippet shows part of the local XML file relevant to the node migration:

<?xml version="1.0" encoding="UTF-8" ?>
<data>
<udm_people>
<unique_id>1</unique_id>
<name>Michele Metts</name>
<photo_file>P01</photo_file>
<book_ref>B10</book_ref>
</udm_people>
<udm_people>
...
</udm_people>
<udm_people>
...
</udm_people>
</data>

The set of elements containing node data lies two levels deep in the hierarchy. Starting with data at the root and then descending one level to udm_people. Each element of this array is an object with four properties:

• unique_id is the unique identifier for each element returned by the data/udm_people hierarchy.
• name is the name of a person. This will be used in the node title.
• photo_file is the unique identifier of an image that was created in a separate migration.
• book_ref is the unique identifier of a book paragraph that was created in a separate migration.

The following snippet shows the configuration to read a local XML file for the node migration:

source:
  plugin: url
  # This configuration is ignored by the 'xml' data parser plugin.
  # It only has effect when using the 'simple_xml' data parser plugin.
  data_fetcher_plugin: file
  # Set to 'xml' to use XMLReader https://www.php.net/manual/en/book.xmlreader.php
  # Set to 'simple_xml' to use SimpleXML https://www.php.net/manual/en/ref.simplexml.php
  data_parser_plugin: xml
  urls:
    - modules/custom/ud_migrations/ud_migrations_xml_source/sources/udm_data.xml
  # XPath expression. It is common that it starts with a slash (/).
  item_selector: /data/udm_people
  fields:
    - name: src_unique_id
      label: 'Unique ID'
      selector: unique_id
    - name: src_name
      label: 'Name'
      selector: name
    - name: src_photo_file
      label: 'Photo ID'
      selector: photo_file
    - name: src_book_ref
      label: 'Book paragraph ID'
      selector: book_ref
  ids:
    src_unique_id:
      type: integer

The name of the plugin is url. Because we are reading a local file, the data_fetcher_plugin  is set to file and the data_parser_plugin to xml. The urls configuration contains an array of file paths relative to the Drupal root. In the example we are reading from one file only, but you can read from multiple files at once. In that case, it is important that they have a homogeneous structure. The settings that follow will apply equally to all the files listed in urls.

Technical note: Migrate Plus provides two data parser plugins for XML files. xml uses XMLReader while simple_xml uses SimpleXML. The parser to use is configured in the data_parser_plugin configuration. Also note that when you use the xml parser, the data_fetcher_plugin setting is ignored. More details below.

The item_selector configuration indicates where in the XML file lies the set of elements to be migrated. Its value is an XPath expression used to traverse the file hierarchy. In this case, the value is /data/udm_people. Verify that your expression is valid and select the elements you intend to import. It is common that it starts with a slash (/).

fields has to be set to an array. Each element represents a field that will be made available to the migration. The following options can be set:

• name is required. This is how the field is going to be referenced in the migration. The name itself can be arbitrary. If it contained spaces, you need to put double quotation marks (") around it when referring to it in the migration.
• label is optional. This is a description used when presenting details about the migration. For example, in the user interface provided by the Migrate Tools module. When defined, you do not use the label to refer to the field. Keep using the name.
• selector is required. This is another XPath-like string to find the field to import. The value must be relative to the subtree specified by the item_selector configuration. In the example, the fields are direct children of the elements to migrate. Therefore, the XPath expression only includes the element name (e.g., unique_id). If you had nested elements, you could use a slash (/) character to go deeper in the hierarchy. This will be demonstrated in the image and paragraph migrations.

Finally, you specify an ids array of field names that would uniquely identify each record. As already stated, the unique_id field servers that purpose. The following snippet shows part of the process, destination, and dependencies configuration of the node migration:

process:
  field_ud_image/target_id:
    plugin: migration_lookup
    migration: udm_xml_source_image
    source: src_photo_file
destination:
  plugin: 'entity:node'
  default_bundle: ud_paragraphs
migration_dependencies:
  required:
    - udm_xml_source_image
    - udm_xml_source_paragraph
  optional: []

The source for the setting the image reference is src_photo_file. Again, this is the name of the field, not the label nor selector. The configuration of the migration lookup plugin and dependencies point to two XML migrations that come with this example. One is for migrating images and the other for migrating paragraphs.

Migrating paragraphs from a XML file

Let’s consider an example where the elements to migrate have many levels of nesting. The following snippets show part of the local XML file and source plugin configuration for the paragraph migration:

<?xml version="1.0" encoding="UTF-8" ?>
<data>
  <udm_book_paragraph>
    <book_id>B10</book_id>
    <book_details>
      <title>The Definitive Guide to Drupal 7</title>
      <author>Benjamin Melançon et al.</author>
    </book_details>
  </udm_book_paragraph>
  <udm_book_paragraph>
    ...
  </udm_book_paragraph>
  <udm_book_paragraph>
    ...
  </udm_book_paragraph>
</data>

source:
  plugin: url
  # This configuration is ignored by the 'xml' data parser plugin.
  # It only has effect when using the 'simple_xml' data parser plugin.
  data_fetcher_plugin: file
  # Set to 'xml' to use XMLReader https://www.php.net/manual/en/book.xmlreader.php
  # Set to 'simple_xml' to use SimpleXML https://www.php.net/manual/en/ref.simplexml.php
  data_parser_plugin: xml
  urls:
    - modules/custom/ud_migrations/ud_migrations_xml_source/sources/udm_data.xml
  # XPath expression. It is common that it starts with a slash (/).
  item_selector: /data/udm_book_paragraph
  fields:
    - name: src_book_id
      label: 'Book ID'
      selector: book_id
    - name: src_book_title
      label: 'Title'
      selector: book_details/title
    - name: src_book_author
      label: 'Author'
      selector: book_details/author
  ids:
    src_book_id:
      type: string

The plugin, data_fetcher_plugin, data_parser_plugin and urls configurations have the same values as in the node migration. The item_selector and ids configurations are slightly different to represent the path to paragraph elements and the unique identifier field, respectively.

The interesting part is the value of the fields configuration. Taking data/udm_book_paragraph as a starting point, the records with paragraph data have a nested structure. Particularly, the book_details element has two children: title and author. To refer to them, the selectors are book_details/title and book_details/author, respectively. Note that you can go as many level deeps in the hierarchy to find the value that should be assigned to the field. Every level in the hierarchy could be separated by a slash (/).

In this example, the target is a single paragraph type. But a similar technique can be used to migrate multiple types. One way to configure the XML file is having two children. paragraph_id would contain the unique identifier for the record. paragraph_data would contain a child element to specify the paragraph type. It would also have an arbitrary number of extra child elements with the data to be migrated. In the process section, you would iterate over the children to map the paragraph fields.

The following snippet shows part of the process configuration of the paragraph migration:

process:
  field_ud_book_paragraph_title: src_book_title
  field_ud_book_paragraph_author: src_book_author

Migrating images from a XML file

Let’s consider an example where the elements to migrate have more data than needed. The following snippets show part of the local XML file and source plugin configuration for the image migration:

<!--?xml version="1.0" encoding="UTF-8" ?-->
<data>
  <udm_photos>
<photo_id>P01</photo_id>
<photo_url>https://agaric.coop/sites/default/files/pictures/picture-15-1421176712.jpg</photo_url>
<photo_dimensions>
      <width>240</width>
      <height>351</height>
    </photo_dimensions>
  </udm_photos>
  <udm_photos>
    ...
  </udm_photos>
  <udm_photos>
    ...
  </udm_photos>
</data>

source:
  plugin: url
  # This configuration is ignored by the 'xml' data parser plugin.
  # It only has effect when using the 'simple_xml' data parser plugin.
  data_fetcher_plugin: file
  # Set to 'xml' to use XMLReader https://www.php.net/manual/en/book.xmlreader.php
  # Set to 'simple_xml' to use SimpleXML https://www.php.net/manual/en/ref.simplexml.php
  data_parser_plugin: xml
  urls:
    - modules/custom/ud_migrations/ud_migrations_xml_source/sources/udm_data.xml
  # XPath expression. It is common that it starts with a slash (/).
  item_selector: /data/udm_photos
  fields:
    - name: src_photo_id
      label: 'Photo ID'
      selector: photo_id
    - name: src_photo_url
      label: 'Photo URL'
      selector: photo_url
  ids:
    src_photo_id:
      type: string

The following snippet shows part of the process configuration of the image migration:

process:
  psf_destination_filename:
    plugin: callback
    callable: basename
    source: src_photo_url

The plugin, data_fetcher_plugin, data_parser_plugin and urls configurations have the same values as in the node migration. The item_selector and ids configurations are slightly different to represent the path to image elements and the unique identifier field, respectively.

The interesting part is the value of the fields configuration. Taking data/udm_photos as a starting point, the elements with image data have extra children that are not used in the migration. Particularly, the photo_dimensions element has two children representing the width and height of the image. To ignore this subtree, you simply omit it from the fields configuration. In case you wanted to use it, the selectors would be photo_dimensions/width and photo_dimensions/height, respectively.

XML file location

Important: What is described in this section only applies when you use either (1) the xml data parser or (2) the simple_xml parser with the file data fetcher.

When using the file data fetcher plugin, you have three options to indicate the location to the XML files in the urls configuration:

• Use a relative path from the Drupal root. The path should not start with a slash (/). This is the approach used in this demo. For example, modules/custom/my_module/xml_files/example.xml.
• Use an absolute path pointing to the XML location in the file system. The path should start with a slash (/). For example, /var/www/drupal/modules/custom/my_module/xml_files/example.xml.
• Use a fully-qualified URL to any built-in wrapper like http, https, ftp, ftps, etc. For example, https://understanddrupal.com/xml-files/example.xml.
• Use a custom stream wrapper.

Being able to use stream wrappers gives you many more options. For instance:

• Files located in the public, private, and temporary file systems managed by Drupal. This leverages functionality already available in Drupal core. For example: public://xml_files/example.xml.
• Files located in profiles, modules, and themes. You can use the System stream wrapper module or apply this core patch to get this functionality. For example, module://my_module/xml_files/example.xml.
• Files located in AWS Amazon S3. You can use the S3 File System module along with the S3FS File Proxy to S3 module to get this functionality.

Migrating remote XML files

Important: What is described in this section only applies when you use the http data fetcher plugin.

Migrate Plus provides another data fetcher plugin named http. Under the hood, it uses the Guzzle HTTP Client library. You can use it to fetch files using any protocol supported by curl like http, https, ftp, ftps, sftp, etc. In a future blog post we will explain this data fetcher in more detail. For now, the udm_xml_source_node_remote migration demonstrates a basic setup for this plugin. Note that only the data_fetcher_plugin, data_parser_plugin, and urls configurations are different from the local file example. The following snippet shows part of the configuration to read a remote XML file for the node migration:

source:
  plugin: url
  data_fetcher_plugin: http
  # 'simple_xml' is configured to be able to use the 'http' fetcher.
  data_parser_plugin: simple_xml
  urls:
    - https://sendeyo.com/up/d/478f835718
  item_selector: /data/udm_people
  fields: ...
  ids: ...

And that is how you can use XML files as the source of your migrations. Many more configurations are possible when you use the simple_xml parser with the http fetcher. For example, you can provide authentication information to get access to protected resources. You can also set custom HTTP headers. Examples will be presented in a future entry.

XMLReader vs SimpleXML in Drupal migrations

As noted in the module’s README file, the xml parser plugin uses the XMLReader interface to incrementally parse XML files. The reader acts as a cursor going forward on the document stream and stopping at each node on the way. This should be used for XML sources which are potentially very large. On the other than, the simple_xml parser plugin uses the SimpleXML interface to fully parse XML files. This should be used for XML sources where you need to be able to use complex XPath expressions for your item selectors, or have to access elements outside of the current item element via XPath.

What did you learn in today’s blog post? Have you migrated from XML files before? If so, what challenges have you found? Did you know that you can read local and remote files? Did you know that the data_fetcher_plugin configuration is ignored when using the xml data parser? Please share your answers in the comments. Also, I would be grateful if you shared this blog post with others.

This blog post series is made possible thanks to these generous sponsors. Contact us if your organization would like to support this documentation project, whether it is the migration series or other topics.

Next: Adding HTTP request headers and authentication to remote JSON and XML in Drupal migrations

This blog post series, cross-posted at UnderstandDrupal.com as well as here on Agaric.coop, is made possible thanks to these generous sponsors: Drupalize.me by Osio Labs has online tutorials about migrations, among other topics, and Agaric provides migration trainings, among other services.  Contact Understand Drupal if your organization would like to support this documentation project, whether it is the migration series or other topics.

See the full size of the mobile design for Vulk's home page.

Respetar su privacidad y ser responsable con los datos que recopilamos de usted es de suma importancia para nosotros. No usaremos ni compartiremos su información con nadie, excepto como se describe en esta política de privacidad.

Recopilación y uso de la información.

Si elige dejar un comentario o un mensaje privado a través de nuestro formulario de contacto, es posible que le solicitemos que nos proporcione cierta información de identificación personal, incluido su nombre y dirección de correo electrónico. La información que recopilamos se utilizará para contactarlo o identificarlo.

Dato de registro

Queremos informarle que cada vez que visite nuestro sitio web, recopilamos la información que su navegador nos envía, que se llama Datos de registro. Estos datos de registro pueden incluir información como la dirección del Protocolo de Internet ("IP") de su computadora, la versión del navegador, las páginas de nuestro sitio web que visite, la fecha y hora de su visita, el tiempo dedicado a esas páginas y otras estadísticas.

Cookies

No utilizamos cookies para los visitantes de nuestro sitio. (Las cookies son archivos con una pequeña cantidad de datos que se utilizan comúnmente como un identificador único anónimo).

Proveedores de servicio

Empleamos una compañía de terceros, Google, para ayudarnos a analizar cómo se utiliza nuestro sitio web. Sólo se recopilan datos anónimos.

Hotjar nos ayuda a proporcionar a nuestros usuarios finales una mejor experiencia y servicio, así como a diagnosticar problemas técnicos y analizar las tendencias de los usuarios. Lo más importante es que a través de los servicios de Hotjar, la funcionalidad del sitio puede mejorarse, haciéndolos más fáciles de usar, más valiosos y más fáciles de usar para los usuarios finales.

Puede optar por no permitir que Hotjar recopile su información cuando visite un sitio habilitado para Hotjar en cualquier momento visitando nuestra página de exclusión y haciendo clic en "Deshabilitar Hotjar" o habilitando No rastrear (DNT) en su navegador.

Cambios a esta política de privacidad

Podemos actualizar nuestra Política de Privacidad de vez en cuando. Por lo tanto, le recomendamos que revise esta página periódicamente para cualquier cambio. Le notificaremos cualquier cambio mediante la publicación de la nueva Política de privacidad en esta página. Estos cambios entrarán en vigencia inmediatamente después de que se publiquen en esta página.

Contáctenos

Si tiene alguna pregunta o sugerencia sobre nuestra política de privacidad, no dude en ponerse en contacto con nosotros.