Blog

Always run the desired PHP version when your hosting solution has multiple versions available

2023 February 03

Always run the desired PHP version when your hosting solution has multiple versions available

"Oh, you wanted that PHP?"

At Agaric, we perform a lot of Drupal upgrades. These very often involve transitioning away from older versions of PHP. Even when your hosting service provides multiple versions of PHP, you can still run into issues activating the appropriate one for each site: whether that's within the web server, at the command line, or via Drush (or other tools). In this blog post, we'll be providing remedies for all of these cases. Most content applies to any PHP application.

Only Section 3 is particularly Drupal/Drush specific, but still might give you a hint about running remote commands that use ssh from behind their shiny CLI.

TLDR: just give me what works, NOW (please?)

Since you asked so politely: if you are already familiar with hosting options, bash shell configuration, and Drush, this somewhat lengthy article can be summed up quite quickly, really. For those that might not be pro's in any one of these, you can take this as the quick intro, and keep reading to learn why these situations exist, as well as the detailed fix.

If you are on a hosting provider, they normally will have a UI to select the desired PHP version. If you are doing your own hosting, install Ondrej Sury's PHP PPA/Apt repo, and install PHP-FPM (not mod_php). Adjust your SetHandler (Apache), or fastcgi_pass (NGINX) in each site configuration to the socket for the pool of the version of PHP you want.
Configure a local ~/bin folder and place a link to the php version you want (e.g. ln -s /usr/bin/php8.1 ~/bin/php) there. Be certain ~/bin is also in your PATH.
When using Drush aliases to run commands on remote instances, this still won't work until you add an env-vars: key to your site.yml file and place a PATH in there.

Jump to the "Part" below if you only need help with one or more of these.

Part 1: All things hosting (well, at least Apache or NGINX, plus FPM)

Let's start where most people do: setting up the hosting itself. In most cases, a managed hosting provider will have some way to select the appropriate version of PHP for your individual site. If that's the case - pick your target, and you should be good to go (move on to Part 2, you lucky dog)! If you do your own hosting, there will be some additional steps to take, which I'll give an overview of, and some additional resources to get you over this first hurdle.

If you are running "A-PAtCHy" web server (possible name change coming?) you will not be able to use mod_php for your PHP duties, as this method does not allow the web server to directly serve content of different virtual hosts using different versions of PHP. Instead, I recommend using PHP's "FastCGI Process Manager" service - aka FPM. This is a stand-alone service that Apache and NGINX will speak to using new-age technology from 1996 called FastCGI. It's still technically CGI, only, like, Fast (seriously, it works really well). Your web server hands off to this service with it's related FastCGI/proxy module.

The process is quite similar for both web servers, and an article over at Linode covers the basics of this method for each, but wait! Finish reading at least this paragraph before you jump over there for both a caveat emptor and then some Debian specific derivations, if you need those (the article is Ubuntu-specific). In the article, they utilize the excellent PHP resources offered by Ondřej Surý. From this PPA/Debian APT resource, you can run concurrent installations of any of the following PHP versions (listed as of this writing): 5.6, 7.0, 7.1, 7.2, 7.3, 7.4, 8.0, 8.1, and 8.2. Do keep in mind, however, that versions prior to 8.0 are [per php.net] now past their supported lifetime and no longer actively developed (see also this FAQ for more details). Debian-specific instructions for setting up this repository in apt (as opposed to Ubuntu PPA support) are also found within Ondřej's instructions. The remainder of the Linode process should still apply with that one change. OK, run along and get the PHP basics talking to your web server. If you'll be running Drupal (why wouldn't you?) then you'll want to ensure you have the version-specific php modules that it requires (this is for Drupal 9+, but links to earlier revisions also). I'll wait...

Assuming you have now progressed as far as having both versions of PHP you need installed, and followed the article from Linode above or whatever your favorite substitute source was, you likely noticed the special sauce that connects the web server to a particular PHP FPM service. In Apache, we have: SetHandler "proxy:unix:/var/run/php/php8.0-fpm.sock|fcgi://localhost", and in NGINX flavor, it's: fastcgi_pass unix:/var/run/php/php8.0-fpm.sock; These directives point to the Unix socket as defined for in the default PHP "pool" for the particular version. For reference, these are defined in /etc/php/{version}/fpm/pool.d/www.conf and therein will look like: listen = /var/run/php/php8.1-fpm.sock. So - all that's necessary to select your PHP version for your web server is to point to whichever socket location for the version of PHP you want.

The Linode article does not go into handling multiple host names, and I won't go too deep here either as I've already navigated headlong into a bit of a scope-creep iceberg. The quick-and-dirty: for Apache, add another site configuration (as in, add another your_site.conf in /etc/apache/sites-available, and link to it from sites-enabled) repeating the entire VirtualHost and everything inside it, however, use a different listen port, or the same port, and add the ServerName directive to specify the unique DNS name. Likewise, with NGINX, except you here you repeat the full server block in another configuration, and changing the listen and/or server_name bits. Oh yeah - you'll probably be changing the folder location of the Drupal installation in there too, that should definitely help reduce some confusion.

Phew - we should have the web server out of the way now!

Part 2: BASH-ing the `php` command

Next up: PHP on your command line. Here, I'm referring to what you get when you type php on your command line once logged in (via ssh) as the user that manages the web site. In this section, I'm assuming that per best-practices, there are different users for each site. This method does not help much if you have but one user...though I guess it can help if they're both using the same non-default version of php, in contrast to, say, other users on the server.

Which PHP is which?

When a server has multiple versions of PHP, only one of them at a given time will ever live at the path /usr/bin/php. Ordinarily, this is what you get when you type just php at the command line. This also is what you'll get whenever you run a file with a shebang of #!/usr/bin/env php, meaning if you run drush (or wp-cli, for our Wordpress friends), you'll get whatever PHP is found there as well. You can run which php if you'd like to see where php is found.

At this point, definitely check php --version. If you are getting the version you want, you're are done with this article! Well, maybe not - you just want to switch to the account where you do require a different version of PHP than this gives you.

So, php --version gives one version, but there should be multiple PHP executables on the system now, right? (You should have installed multiple, or else have multiple versions available at this point). So where are those? These can be directly executed by running them using their versions in their name. For example php8.1 or php7.4. So, what is happening here that we just get "one proper php"? Well, a couple things.

Environmental conditions...

What, the smog? No...well, yes, that's a problem, but some good things happened in 2022. In this instance, our first issue comes from an environment variable. In particular: the venerable PATH, which contains a list of locations that the shell will use to look for a given executable as you enter a command name (or, again, as specified by a shebang). The PATH variable is a colon-delimited list, typically looking about like this: /usr/local/bin:/usr/bin:/bin:. The shell simply looks for the first occurrence of your command as it peeks in each directory in sequence (left-to-right). Is there a /usr/local/bin/php? That's what you'll get. If not, how about /usr/bin/php? And so on, until it finds one, or else you've mistyped pph and end up with command not found instead. You can see your path with echo $PATH (or try which agaric. I'm guessing you won't have an agaric program, this will tell you where it looked for it when it fail to find one).

What's the 'alternative': links all the way down

The second part of this equation is what is going on with the /usr/bin/php that was found. This alleged "The PHP" is actually a soft link to the current system-level default version of PHP that's installed. You can see how this situation is resolved with a command such as readlink -f /usr/bin/php. This command basically says "read the symbolic link (recursively, due to the -f, try it again without the -f!) and show what it's [ultimately] pointing to". This link (and those it links to) come from an "alternatives" system used by Debian-like systems that connects such things as the canonical name of an executable to a specific installed version. You can learn more about how how this is set up (for PHP, anyway) from...you guessed it: Ondřej's FAQ.

So...the fix then?

Now, where we have multiple versions of PHP installed, it's generally impractical to change all the shebang lines to something else, but that is technically one way to do things. We're also assuming you want to use multiple versions simultaneously here - so updating via the alternatives system isn't a great option either - if you can even do that. There is a simple method to make this work, even as an unprivileged user: make your own link called php and make sure the shell can find it in the PATH.

What we will do, is create our own ~/bin folder, and make our link there. Then, we just make sure that ~/bin is in our path, and comes before other locations that have a php file. There's no shortage of places for customizing the PATH (and bash, generally), and quite frankly, since I'm not positive what the canonical location is, I'll happily follow the Debian manual which says ~/.bashrc. The particular file you'll want to use can be influenced by the type of shell you request (a login vs non-login, and interactive vs non-interactive). In the manual, part of their example looks like this:


# set PATH so it includes user's private bin if it exists
if [ -d ~/bin ] ; then
  PATH="~/bin${PATH+:$PATH}"
fi
export PATH

Curious what that odd-looking ${PATH+:$PATH} syntax is about? Most people just refer to PATH as $PATH when they want it, like this PATH=~/bin:$PATH, right? Well, yes, and that will probably work just fine, but that weird reference does have some smarts. These are something called parameter expansions. Note that in proper bash parlance, the thing we've been calling a variable this whole time is referred to as a parameter. Go figure...that certainly didn't help me find this reference documentation. If you are interested in shell programming (which I clearly think everyone is, or should be) these can be very helpful to know. You'll bump into these when performing various checks and substitutions on varia-er, parameters. Check out the bash documentation to figure this one out.

OK, this looks good! Let's go ahead and add that to the ~/.bashrc file in your ssh user's home folder. Now, you either need to source the updated file (with . ~/.bashrc), or just reconnect. Sourcing (abbreviated with that previous ., otherwise spelled out as source ~/.bashrc) essentially executes the referenced file in a way that can it can modify the current shell's context. If you were to just run ~/.bashrc without sourcing it, it happily sets up an environment, but all that gets wiped out when the script ends.

Now, let's get moving with this plan again. Make a bin folder in our ssh user's home folder: mkdir ~/bin. Finally, link the php version you want in there. Here, I'll do php8.1: ln -s /usr/bin/php8.1 ~/bin/php. Voila! Now when you run php --version, you'll get PHP 8.1!

Another diversion, really?

For the impatient: reconnect and skip to the paragraph. For the curious (or when you have time to come back): it turns out our shell has a bit of memory we never expected it to. If you've typed php sometime earlier in your session, and bash last found that it was the one in /usr/bin/php that came up first, it's now remembered that so it doesn't have to find it again. While you can just re-login - again! - you might also wan to take the reins of your system and try typing hash -d php (see help hash to learn what that does - the help command covers shell built-in functionality, like hash). At last, php really works the way we wanted! No more odd little shell corners hiding dusty references on us.

OMG, that works?

Finally...despite my droning on, we're making progress! At this point, when you call upon your drush status, it should actually run without errors (some things under php7.x just don't work now - as expected) and show that it's using the correct php version.

Part 3: Drush does speak SSH, but not YOUR SSH.

The Drush boffins graced us with doodads called site aliases that allow us to readily send commands to various sites we control. If you don't know about those, you'll have to read up on them first. The prolific Moshe Weitzman (along with some 310 contributors, and counting) didn't give us these because they were getting bored after 20 years of Drupal; They're pretty essential to working with Drupal effectively.

Assuming you have a grasp of aliases under your belt, let's try a Drush command from our local environment to the remote system: drush @test status. OK - something is wrong here. My Drush just said I'm still using php7.4 and not my beloved 8.1...again! Does Drush have a memory too? No, it just doesn't care how you want your ssh sessions to work; those are left to you and Drush does it's own thing. The ssh connection that Drush opens has it's own context set-up, unlike the one we have when we've ssh'd in interactively. Thankfully, there's a quick fix for this - they key Drush needs to know is how to set the PATH up so it also gets our targeted PHP version. Let's add our modified PATH to the site aliases configuration, so Drush also knows what the cool kids are doing.

live: host: example.com paths: drush-script: /home/example-live/vendor/bin/drush env-vars: PATH: /home/example-live/bin:/usr/bin:/bin root: /home/example-live/web uri: 'https://example.com/' user: example-live

Note the specified PATH of /home/example-live/bin:/bin:/usr/bin places the bin directory in home at the beginning again. At last, when we run drush @test status, it's telling us the PHP it uses is the one that we use. We can share, Drush. We made it!

Thanks for stopping by!

That wraps it up for this one. Hopefully you now feel a little more confident you have some tools helping you master your environment. The version of PHP you need is now at your command. Now you can go get busy with your composer updates and other TODO's. By the way, if all this configuration headache is just not for you - check out our hosted Drutopia platform. We run an instance of this distribution, and provide all these niceties as part of our hosting setup, so reach out if interested. Either way, thanks for coming by!

February 3, 2023

Oh yeah!

I'm back! Because I knew this blog post wasn't quite long enough already...certainly not because it occurred to me that I left off another alternative I've used in a pinch.

The one-off

If you want to run a command using a different version of php, rather than rely on the shebang, you can always just call the desired script as an argument to php itself. When you run, e.g. drush status, your shell sees the #! on the first line, and uses that as the default interpreter for the script. In essence, your command is translated into /usr/bin/env php drush status.

Rather than accept this default, you can specify the specific version on your command line, such as php8.1 drush status. In this case the #! is just ignored. Unfortunately, this method can only get you so far - particularly with drush. Drush often figures out what your command requires and in turn will spawn other php scripts to do the dirty work. These spawned scripts are unaware of the interpreter that drush itself was started with.

How to filter a view by content that references the current node in modern Drupal

2023 January 26

How to filter a view by content that references the current node in modern Drupal

References and contextual filters

It is often enough throughout the journey of building websites that you will desire to create very specific lists of content. In my current endeavor, I am working in Drupal 9 with 2 content types: Office location and Event . Each Event occurs at an Office location, which is indicated by an Office location entity reference field on the Event. Note that any info in this blog will also be relevant to Drupal 8, Drupal 10, and beyond.

Here is the user story that we will be working through:
As a site visitor, when I view an Office location node, I should see a list of Events that are occurring at the location represented by the current node, so that I have an easy way to learn about events at locations relevant to me.

To keep this short and sweet, I am going to assume that you have already created a view. However, all views are not created equal, so if you are not seeing the options that you are looking for while editing your view, maybe try creating a new one. We will be working with a block display—other options are possible, but this is the standard "Drupal approach".

Once you are editing a block display for your view, open the Advanced dropdown on the right side. Here is what you will see:

Steps

1, Add a relationship to content referenced by your entity reference field

Go ahead and click the Add button next to Relationships.

In my case I am looking for Content referenced from field_office_location. This is the entity reference field on my Event content type, and is the field that my view will use for comparison while querying data.

Click add and configure relationships and then Apply on the next screen.

2. Filter your view results by content that references the current node id

Click the Add button next to Contextual Filters

Select ID and click Add and configure contextual filters.

The red boxes above indicate what needs to be done. Be sure to select the relationship you added to the Relationship field. This will allow the contextual filter to use the value of the Event's entity reference field for comparison.

Click Provide default value and select Content ID from URL in the dropdown that appears. Note that if your nodes have URL aliases that do not include the content ID, Drupal will still provide the content ID to your contextual filter. Now your view will know which content to display by comparing the current node ID with the the values in the entity reference field of each listing.

Lastly, Click Apply and do not forget to save your view.

The End

Congratulations! Now we know how to filter our view by content that references the current node. You probably still need to place your block and configure which nodes it should or should not display on, but I think that is a "How to" for another day. Have fun Drupaling!

The development of Therapy Fidelity

2023 January 24

The development of Therapy Fidelity

In support of evidence-based therapy

Therapy Fidelity is a mobile interface that displays 132 sequential steps of cognitive behavioral therapy (CBT) for treating posttraumatic stress disorder (PTSD) to improve the fidelity of following the treatment protocol.

The Challenge

Leaders in the field of psychotherapy have been encouraging psychotherapists to adopt evidence-based treatments such as CBT for 20 years but it has been largely unsuccessful. Therapists have been unwilling and unable to rigorously follow CBT protocols with fidelity. Therapists cannot memorize all 132 steps, and patients have very limited access to the steps. As a result, treatment of patients suffers.

Child psychiatrist Michael Scheeringa, MD identified a solution for this problem—a mobile interface that displays a CBT treatment protocol and that can be viewed by therapists and patients simultaneously. Through the guidance of Dr. Scheeringa and in partnership with Tulane University, Agaric developed the Therapy Fidelity web app.

Our Approach

Designed For Use During a Therapy Session

When a patient attends their therapy session, it is far more likely that they arrive with a mobile phone than with a laptop. The need for a seamless mobile experience was one major reason we decided to build Therapy Fidelity as a single page application using React.js. Single page applications are great for reducing the number of network calls made to the host server since they do not need to request markup templates every time an end-user clicks to a different part of the site.

Consequently, most of the network calls made by the app are made to read or write data to one of several third party APIs. We worked to mitigate these, too, by using the React Query library, a powerful and configurable tool for fetching, caching, and updating data in React. The result is an app that clearly indicates when data is loading, allows the user to continue navigating the app while update requests are being processed, indicates the results of those requests once they resolve, and prevents redundant fetching of data that has already been cached—in other words, a fast, smooth, and intuitive end-user experience, even on a limited-bandwidth device. One additional benefit of the data caching—many third party APIs will bill per API call after a limited number of requests; without caching that data in the browser, these API services could get expensive fast.

Simple, Consistent, and Flexible Design Elements with Tailwind UI/CSS

Throughout the building process, we looked to Tailwind UI for example components. By doing so, we made sure that the app has a consistent look and feel and is responsive so that it looks good on devices of all screen sizes. As this app mostly functions as a utility dashboard, the pre-designed components sufficed for the apps aesthetic needs, meaning that Tailwind UI saved us a lot of time on design work. Often we had to make minor adjustments to the markup so that it would suit our specific needs, and very seldom did we need to add any custom classes to our CSS stylesheet, which incidentally results in removing the often expensive overhead of building and maintaining our CSS architecture. Again, this is great for an app with simple aesthetic needs but which also needs re-usable and flexible responsive components.

HIPAA Certified Storage with TrueVault

In order to facilitate a quickest time-to-market, we opted to utilize an existing HIPAA certified provider for the storage, retrieval, and access controls/monitoring of protected data. TrueVault provides a JavaScript library for communicating with their RESTful backend. Their backend includes provisions for dealing with all aspects of the data, such as authentication, authorization, logging, and so-on. We were able to use their Vault-based data model to separate patient records into individual silo's per-therapist, and use their group-based security to set nearly all relevant permissions for the app's use. TrueVault certainly comes with its limitations and challenges, but it sure would have taken quite a bit more time and resources to build and certify a custom storage system.

Automated Quality Assurance using Typescript and React Testing Library

Additionally, we needed to ensure that each client's data is stored and retrieved accurately. TypeScript, which provides a type system for JavaScript, helped to ensure that as our application passed data around, it was of the expected type. Using a typed language also reduces the possibility for software errors generally. Beyond using good tools to write the application, checks must be in place to ensure that as it evolves, things do not break. Here, we leveraged React Testing framework and other tools to walk through the functionality of the application and ensure that everything continued to function correctly, and even stored the correct data; By mocking (simulating) the TrueVault service we could inspect what the application would be storing there directly.

The Delicate Balance of Computing Large Sets of Sensitive Data

While the majority of permissions available to a logged in client were sufficient, there were cases where privileged tasks needed to be performed. Therapy Fidelity's People Like Me system is a great example. People Like Me involves a multi-step form which upon completion shows how many people with similar conditions were able to significantly improve their condition following the successful completion of a 3 month adherence to therapy protocol:

First, a patient selects for comparison the results of a single survey they have completed, consisting of numerical indicators for several psychological conditions (anxiety, depression, PTSD, etc.)

Second, they configure which parameters they would like to match on. For example, a patient could choose to only see how many people who had shared a similar pre-therapy anxiety score were able to improve their condition following therapy.

Finally, they view the results.

Now it must be acknowledged that this system is parsing through and accessing patient data in real time. Of course, it would be a breach of patient privacy law for the patient to have such access to all of this other patient data, so we made sure that all data was scrubbed before reaching storage to reduce any personally identifiable information, and that computations occurred securely on a server before serving back results to the client.

Rather than utilize something like a full VPS server that performed these functions for us, we were able to leverage functions-as-a-service to provide the specific privileged function access, specifically AWS Lambda functions. This helped us keep the surface area of the application as low as possible for security purposes, while providing server-side style functionality. The language we chose for these was Go, for its performance, excellent built-in framework, and because we were able to build for our needs with very minimal additional libraries, which again helps mitigate security concerns.

While these functions-as-a-service were used several times throughout the site, it will be hard to forget Dr. Scheeringa's excitement about the realization of the People Like Me tool.

What's Next?

Therapy Fidelity was an excellent development experience—both for the thrill of developing a single page app and for carrying on Agaric's history of building tools for medical and scientific communities. As always, we are looking forward to our next opportunity to engage in these communities and build something wonderful.

For any clinicians who happen to be reading this and are interested in using the Therapy Fidelity app for their own practice, Scheeringa Mind offers it as a service and provides a 30-day free trial. There are also plenty of video tutorials on how to use the app, which might help to preview just what you might be getting into.

As Scheeringa Mind Company grows and Therapy Fidelity sees more use, we will be sure to publish any updates or major achievements. We are anticipating great strides in research on CBT protocol and look forward to seeing the results.

Recommendations for social media as Twitter melts down: Host your own community

2022 November 18

Recommendations for social media as Twitter melts down

Host your own community!

As Elon Musk destroys Twitter, a lot of clients have asked about alternative social media, especially 'Mastodon'— meaning the federated network that includes thousands of servers, running that software and many other FLOSS applications, all providing interconnecting hubs for distributed social media. Agaric has some experience in those parts, so we are sharing our thoughts on the opportunity in this crisis.

In short: For not-for-profit organizations and news outlets especially, this is a chance to host your own communities by providing people a natural home on the federated social web.

Every not-for-profit organization lives or dies, ultimately, based on its relationship with its supporters. Every news organization, it's readers and viewers.

For years now, a significant portion of the (potential) audience relationship of most organizations has been mediated by a handful of giant corporations through Google search, Facebook and Twitter social media.

A federated approach based on a protocol called ActivityPub has proven durable and viable over the past five years. Federated means different servers run by different people or organizations can host people's accounts, and people can see, reply to, and boost the posts of people on the other servers. The most widely known software doing this is Mastodon but it is far from alone. Akkoma, Pleroma, Friendica, Pixelfed (image-focused), PeerTube (video-focused), Mobilizon (event-focused), and more all implement the ActivityPub protocol. You can be viewing and interacting with someone using different software and not know it— similar to how you can call someone on the phone and not know their cellular network nor their phone model.

The goal of building a social media following of people interested in (and ideally actively supporting) your organization might be best met by setting up your own social media.

This is very doable with the 'fediverse' and Mastodon in particular. In particular, because the number of people on this ActivityPub-based federated social web has already grown by a couple million in the past few weeks— and that's with Twitter not yet having serious technical problems that are sure to come with most of its staff laid off. With the likely implosion of Twitter, giving people a home that makes sense for them is a huge service in helping people get started— the hardest part is choosing a site!

People fleeing Twitter as it breaks down socially and technically would benefit from your help in getting on this federated social network. So would people who have never joined, or long since left, Twitter or other social media, but are willing to join a network that is less toxic and is not engineered to be addictive and harmful.

Your organization would benefit by having a relationship with readers that is not mediated by proprietary algorithms nor for-profit monopolies. It makes your access on this social network more like e-mail lists— it is harder for another entity to come in between you and your audience and take access away.

But the mutual benefits for the organization and its audience go beyond all of this.

When people discuss among one another what the organization has done and published, a little bit of genuine community forms.

Starting a Mastodon server could be the start of your organization seeing itself as not only doing good works or publishing media, but building a better place for people to connect and create content online.

The safety and stability of hosting a home on this federated social network gives people a place to build community.

But organizations have been slow to adopt, even now with the Twitter meltdown. This opens up tho opportunity for extra attention and acquiring new followers.

Hosting the server could cost between $50 to $450 a month, but this is definitely an opportunity to provide a pure community benefit (it is an ad-free culture) and seek donations, grants, or memberships.

The true cost is in moderation time; if volunteers can start to fill that you are in good shape. A comprehensive writeup on everything to consider is here courtesy the cooperatively-managed Mastodon server that Agaric Technology Collective chose to join at social.coop's how to make the fediverse your own.

You would be about the first for not-for-profit or news organizations.

You would be:

giving people a social media home right when they need it
literally owning the platform much of your community is on

And it all works because of the federation aspect— your organization does not have to provide a Twitter, TikTok, or Facebook replacement yourselves, you instead join the leading contender for all that.

By being bold and early, you will also get media attention and perhaps donations and grants.

The real question is if it would divert scarce resources from your core work, or if the community-managing aspects of this could bring new volunteer (or better, paid) talent to handle this.

Even one person willing to take on the moderator role for a half-hour a day to start should be enough to remove any person who harasses people on other servers or otherwise posts racist, transphobic, or other hateful remarks.

Above all, your organization would be furthering your purpose, through means other than its core activities or publishing, to inform and educate and give people more capacity to build with you.

Not surprisingly, Drupal has already figured this out!

Image credit: Karlheinrichpasch, CC BY-SA 4.0, via Wikimedia Commons.

How to add an audio player to a content node in Drupal 9

2022 September 20

How to add an audio player to a content node in Drupal 9

Getting an audio player to show on your content nodes only takes a few steps. However, like many things in Drupal, knowing what those few steps are can be a bit of a challenge. This short guide will provide a basic introduction for anyone looking to use this core functionality on their site.

Step 1: Enable the Media module

The media module is available in Drupal core, and just needs to be enabled.

Step 2: Create a new Media type

In the admin toolbar, navigate to Structure > Media types > Add media type
You may or may not have specific needs that determine how you name your media type. I simply named mine Audio file, and then selected Audio file for the Media source selection field. Don't forget to click save.

Step 3: Create an entity reference field on the entity type that should contain your audio player

In the most basic case, you will add a field to the content type where you want an audio player. When you get to the field type drop-down, select Media (under Reference). Of course, label your field according to your needs.

note: If you are using the paragraphs module, you might choose to create a new paragraph type and add a field there instead.

Click Save and continue

Click Save field settings

Now you will be directed to the Edit tab for your new field. Select your desired settings. The important bit here is that you scroll to the REFERENCE TYPE field set, and click the checkbox next to your new media type (Audio file in my case).

Scroll to the bottom and click Save settings

If you added this field directly to your content type, then you should be all set.
If you are using paragraphs, don't forget to enable your new paragraph type on the paragraph field of your content type.

Drupal 9 Component Plugin ContextException "not a valid context" after CTools, Symfony update

2022 July 29

Drupal 9 Component Plugin ContextException "not a valid context"

after CTools, Symfony update

Upgrading to CTools 3.10.0 and/or various Symfony upgrades can break the simple task of adding content on your Drupal site.

Drupal\Component\Plugin\Exception\ContextException: The context is not a valid context. in Drupal\Core\Executable\ExecutablePluginBase->getContextDefinition() (line 184 …

Update: Seems confirmed this is CTools, see critical bug issue.

And this fixed it:

git revert 0fbbbaf
[main 7be1cd2] Revert fatal error standard code update "Update composer lock"
 1 file changed, 163 insertions(+), 220 deletions(-)
saharareporters [main]$ dco install # that is, composer install - dco is my alias for ddev composer
Gathering patches for root package.
Installing dependencies from lock file (including require-dev)
Verifying lock file contents can be installed on current platform.
Warning: The lock file is not up to date with the latest changes in composer.json. You may be getting outdated dependencies. It is recommended that you run `composer update` or `composer update <package name>`.
Package operations: 0 installs, 27 updates, 1 removal
  - Removing drupal/entityqueue (1.2.0)
Deleting web/modules/contrib/entityqueue - deleted
Gathering patches for root package.
Gathering patches for dependencies. This might take a minute.
  - Downgrading drupal/core-composer-scaffold (9.4.4 => 9.4.3): Extracting archive
  - Downgrading symfony/yaml (v4.4.44 => v4.4.43): Extracting archive
  - Downgrading symfony/translation (v4.4.44 => v4.4.41): Extracting archive
  - Downgrading symfony/event-dispatcher (v4.4.44 => v4.4.42): Extracting archive
  - Downgrading symfony/dependency-injection (v4.4.44 => v4.4.43): Extracting archive
  - Downgrading symfony/console (v4.4.44 => v4.4.43): Extracting archive
  - Downgrading symfony/filesystem (v5.4.11 => v5.4.9): Extracting archive
  - Downgrading symfony/config (v4.4.44 => v4.4.42): Extracting archive
  - Downgrading symfony/css-selector (v5.4.11 => v5.4.3): Extracting archive
  - Downgrading symfony/dom-crawler (v4.4.44 => v4.4.42): Extracting archive
  - Downgrading symfony/browser-kit (v4.4.44 => v4.4.37): Extracting archive
  - Downgrading symfony/finder (v6.1.3 => v6.1.0): Extracting archive
  - Downgrading symfony/validator (v4.4.44 => v4.4.43): Extracting archive
  - Downgrading symfony/serializer (v4.4.44 => v4.4.43): Extracting archive
  - Downgrading symfony/routing (v4.4.44 => v4.4.41): Extracting archive
  - Downgrading symfony/mime (v5.4.11 => v5.4.10): Extracting archive
  - Downgrading symfony/http-foundation (v4.4.44 => v4.4.43): Extracting archive
  - Downgrading symfony/process (v4.4.44 => v4.4.41): Extracting archive
  - Downgrading symfony/var-dumper (v5.4.11 => v5.4.9): Extracting archive
  - Downgrading symfony/debug (v4.4.44 => v4.4.41): Extracting archive
  - Downgrading symfony/error-handler (v4.4.44 => v4.4.41): Extracting archive
  - Downgrading symfony/http-kernel (v4.4.44 => v4.4.43): Extracting archive
  - Downgrading drupal/migmag (1.8.1 => 1.8.0): Extracting archive
  - Downgrading symfony/string (v6.1.3 => v6.1.2): Extracting archive
  - Downgrading drupal/noreferrer (1.14.0 => 1.13.0): Extracting archive
  - Downgrading drupal/ctools (3.10.0 => 3.9.0): Extracting archive
  - Downgrading symfony/mailer (v5.4.11 => v5.4.10): Extracting archive

And a drush cache-reload, of course.

Full error, for search engines:

The website encountered an unexpected error. Please try again later.

Drupal\Component\Plugin\Exception\ContextException: The context is not a valid context. in Drupal\Core\Executable\ExecutablePluginBase->getContextDefinition() (line 184 of core/lib/Drupal/Core/Plugin/ContextAwarePluginTrait.php).

Drupal\Core\Executable\ExecutablePluginBase->getContext(NULL) (Line: 116)
Drupal\Core\Executable\ExecutablePluginBase->getContextValue(NULL) (Line: 91)
Drupal\Core\Entity\Plugin\Condition\EntityBundle->evaluate() (Line: 77)
Drupal\Core\Condition\ConditionManager->execute(Object) (Line: 84)
Drupal\Core\Condition\ConditionPluginBase->execute() (Line: 376)
Drupal\pathauto\Entity\PathautoPattern->applies(Object) (Line: 320)
Drupal\pathauto\PathautoGenerator->getPatternByEntity(Object) (Line: 41)
Drupal\pathauto\PathautoWidget->formElement(Object, 0, Array, Array, Object) (Line: 353)
Drupal\Core\Field\WidgetBase->formSingleElement(Object, 0, Array, Array, Object) (Line: 220)
Drupal\Core\Field\WidgetBase->formMultipleElements(Object, Array, Object) (Line: 111)
Drupal\Core\Field\WidgetBase->form(Object, Array, Object) (Line: 181)
Drupal\Core\Entity\Entity\EntityFormDisplay->buildForm(Object, Array, Object) (Line: 121)
Drupal\Core\Entity\ContentEntityForm->form(Array, Object) (Line: 127)
Drupal\node\NodeForm->form(Array, Object) (Line: 106)
Drupal\Core\Entity\EntityForm->buildForm(Array, Object)
call_user_func_array(Array, Array) (Line: 531)
Drupal\Core\Form\FormBuilder->retrieveForm('node_article_form', Object) (Line: 278)
Drupal\Core\Form\FormBuilder->buildForm(Object, Object) (Line: 97)
Drupal\autosave_form\Form\AutosaveFormBuilder->buildForm(Object, Object) (Line: 73)
Drupal\Core\Controller\FormController->getContentResult(Object, Object)
call_user_func_array(Array, Array) (Line: 123)
Drupal\Core\EventSubscriber\EarlyRenderingControllerWrapperSubscriber->Drupal\Core\EventSubscriber\{closure}() (Line: 564)
Drupal\Core\Render\Renderer->executeInRenderContext(Object, Object) (Line: 124)
Drupal\Core\EventSubscriber\EarlyRenderingControllerWrapperSubscriber->wrapControllerExecutionInRenderContext(Array, Array) (Line: 97)
Drupal\Core\EventSubscriber\EarlyRenderingControllerWrapperSubscriber->Drupal\Core\EventSubscriber\{closure}() (Line: 158)
Symfony\Component\HttpKernel\HttpKernel->handleRaw(Object, 1) (Line: 80)
Symfony\Component\HttpKernel\HttpKernel->handle(Object, 1, 1) (Line: 58)
Drupal\Core\StackMiddleware\Session->handle(Object, 1, 1) (Line: 48)
Drupal\Core\StackMiddleware\KernelPreHandle->handle(Object, 1, 1) (Line: 106)
Drupal\page_cache\StackMiddleware\PageCache->pass(Object, 1, 1) (Line: 85)
Drupal\page_cache\StackMiddleware\PageCache->handle(Object, 1, 1) (Line: 48)
Drupal\Core\StackMiddleware\ReverseProxyMiddleware->handle(Object, 1, 1) (Line: 51)
Drupal\Core\StackMiddleware\NegotiationMiddleware->handle(Object, 1, 1) (Line: 23)
Stack\StackedHttpKernel->handle(Object, 1, 1) (Line: 709)
Drupal\Core\DrupalKernel->handle(Object) (Line: 19)

Drupal 9.4 installation with existing configuration fails because "unable to uninstall the MySQL module"!?

2022 June 17

Here is how to deal with the surprising-to-impossible-seeming error "Unable to uninstall the MySQL module because: The module 'MySQL' is providing the database driver 'mysql'.."

Like, why is it trying to uninstall anything when you are installing? Well, it is because you are installing with existing configuration— and your configuration is out-of-date. This same problem will happen on configuration import on a Drupal website, too. (See update below for those steps!)

Really this error message is a strong reminder to always run database updates and then commit any resulting configuration changes after updating Drupal core or module code.

And so the solution is to roll back the code to Drupal 9.3, do your installation from configuration, and then run the database updates, export configuration, and commit the result.

For example:

git checkout <commit-hash-of-earlier-composer-lock>
composer install
drush -y site:install drutopia --existing-config
git checkout main
composer install
drush -y updb
drush -y cex
git status # Review what is here; git add -p can also help
git add config/
git commit -m "Apply configuration updates from Drupal 9.4 upgrade"

The system update enable_provider_database_driver is the post-update hook that is doing the work here to "Enable the modules that are providing the listed database drivers." Pretty cool feature and a strong reminder to always, always run database updates and commit any configuration changes immediately after any code updates!

Update: Steps on for resolving the unable to uninstall MySQL module error on configuration import

This is what you probably already did, before the drush -y cim failed (luckily, luckily it failed).

composer update
drush -y updb

All that is great! Now continue, not with a config import, but with a config export:

drush -y cex
git status # Review what is here; git add -p can also help
git add config/
git commit -m "Apply configuration updates from Drupal 9.4 upgrade"

Remember, after every composer update and database update, you need to do a configuration export and commit the results— database updates can change configuration, and if you do not commit those, you will undo these intentional and potentially important changes on a configuration import. If you ran into this problem on a configuration import, it is a sign of breakdown in discipline in following these steps!

Every time you bring in code changes with composer update all this must be part and parcel:

composer update
drush -y updb
drush -y cex
git status # Review what is here; git add -p can also help
git add config/
git commit -m "Apply configuration from database updates"

Uniting Visions: Kicking off Thursday 3pm ET planning & building sessions for democratic conversation scaling platform, Visions Unite

2022 June 16

Uniting Visions

Kicking off Thursday 3pm ET planning & building sessions for democratic conversation scaling platform, Visions Unite

Hi friends and collaborators, join us today at 3pm ET (or any subsequent Thursday at 3) as we kick off a series of research, planning, discussion, and building sessions for Visions Unite.

As our primary pro bono project, Agaric is working on Visions Unite, "where people seeking to make the world more whole can share ideas and information and gather the commitment and resources to build power to be the change we need", which a dozen projects have tried to do—what makes this different is sharing power via democratic mass communication.

Here are some initial user stories for Visions Unite.

Help plan and build the interface and underlying technology! (Drupal friends, we have been leaning against Drupal but might do it for the MVP— would love to hear your thoughts for or against.)

Connection info will always be up-to-date at agaric.coop/show (for these sessions we are taking over most of our Show & Tell hour, which is weekly on Thursdays 3pm Eastern).

Good discussion here with a great takeaway. Meet your fellow workers! https://portside.org/2022-05-12/revolutionary-grounds

2022 May 13

Here is a social media microblog-sized post just to say we are syndicating to social.coop Mastodon and Twitter thanks to the Indieweb module!

2022 March 08

It's great to be here, and there, and there.

Thanks Indieweb module for Drupal!

Create and use a custom permission in your module

2021 March 10

Create and use a custom permission in your Drupal module

Define a permission in Drupal code and check access programmatically

You can define your own permissions for the Drupal permissions page (/admin/people/permissions in modern Drupal, Drupal 8, 9, 10, and beyond) and then add conditional options to your code to do different things based on the role of the user and the permissions configured by a site administrator.

Here's how.

Create a modulename.permissions.yml file

This simple file has the permission machine name (lower case with spaces) and a title (Sentence case) with an optional description.

For our module, which has a particularly long name, that file is drutopia_findit_site_management.permissions.yml and its contents are like so:

access meta tab:
  title: 'Access meta tab'
  description: 'Access meta information (author, creation date, boost information) in Meta vertical tab.'

You can repeat lines like these in the same file for as many permissions as you wish to define.

Check for that permission in your code

The process for checking permissions is simply to use a user object if that's handed into your code, or to load the current user if it's not, and use the hasPermission() method which returns TRUE if that user has permission and FALSE if not.

For example, in a form alter in our drutopia_findit_site_management.module file:

/**
 * Implements hook_form_BASE_FORM_ID_alter() for node_form.
 *
 * Completely hide the Meta vertical tab (field group) from people without permission.
 *
 */
function drutopia_findit_site_management_form_node_form_alter(&$form, FormStateInterface $form_state, $form_id) {
  // If the current user has the permission, do not hide the Meta vertical tab.
  if (\Drupal::currentUser()->hasPermission('access meta tab')) {
    return;
  }
  // Code to hide the meta tab goes here, and is only reached if the user lacks the permission. 
  // ...
}

See all this code in context in the Find It Site Management module.

To learn more about defining permissions in modern Drupal, including dynamic permissions, you can see the change record for when the new approach replaced hook_permission().

Why we resist Zoom and choose BigBlueButton for video chat

2021 January 23

We can look at the recent popularity of some widely used platforms like Zoom and ask ourselves some questions as to why we still use them when we know a lot of terrible things about them. Agaric prefers to use a free/libre video chat software called BigBlueButton for many reasons, the first one being the licensing, but there are many reasons.

Zoom has had some major technology failures, which the corporation is not liable to disclose. At one point, a vulnerability was discovered in the desktop Zoom client for MacOS that allowed hackers to start your webcam remotely and launch you into a meeting without your permission. The company posted a note saying that they fixed the issue. Unfortunately, the Zoom source code is proprietary and we are not even allowed to look at it. There is no way for the community to see how the code works or to verify that the fix was comprehensive.

The Zoom Corporation stated early on that the software was encrypted end-to-end (E2EE) from your device to the recipient's device. This was untrue at the time, but the company states that it has been corrected for users on their client app. While it is no longer true that E2EE is unsupported, it does require that you use the proprietary Zoom client for E2EE to work. Without E2EE, any data that is retrieved on its way from your computer to a server can be accessed! The only real security is knowing the operators of your server. This is why Agaric uses trusted sources like MayFirst.org for most of our projects and we have a relationship with our BigBlueButton host. The Intercept also revealed that Zoom users that dial in on their phone are NOT encrypted at all

BigBlueButton does not have a client app and works in your browser, so there is no E2EE. The idea for E2EE is that with it, you "do not have to trust the server operator and you can rely on E2EE" because the model implies that every client has keys that are protecting the transferred data. However: you MUST still use a proprietary client in order to get the benefits of E2EE support, so once again you MUST trust Zoom as you have no permission to examine the app to determine that the keys are not being shared with Zoom.

Of course there is always the fact that hackers work day and night to corrupt E2EE and a Corporation is not obligated to tell you the customer every time there has been a security breach, and this information is usually buried in the terms of service they post - sometimes with a note saying the terms are subject to change and updates. A Corporation is not obligated to tell you, the customer when there has been a security breach" unless any personal information is exposed. There are now mandatory timely disclosure requirements for all states: https://www.ncsl.org/research/telecommunications-and-information-technology/security-breach-notification-laws.aspx ...Can Zoom really be trusted? As with some laws, the fine that is applied is low and affordable and subject to the interpretation of the courts and the status of knowledge your lawyer is privvy to - meaning most Corporations normally have a battery of lawyers to interpret the law and drag the case out until you are... broke.

In the case of BigBlueButton encryption, E2EE would only make sense if there are separate clients using an API to connect to the BBB server so a user does not have to trust the BBB server operator. If the user trusts the server operator, then there would be no need for E2EE." Lesson learned: It is always best practice to know and trust your server hosts as they are the ones that have the keys to your kingdom.

Some technology analysts consider Zoom software to be malware. Within companies that use Zoom, employers are even able to monitor whether or not you are focusing on the computer screen during meetings which seems excessively intrusive. Speaking of intrusive, the Zoom Corporation also shares your data with FaceBook, even if you do not have a FB account - that could be a whole blog in itself, but just being aware of some of the vulnerabilities is a good thing to pass on. Some of the bad stuff remains even if you uninstall the Zoom app from your device! Even though a class action suit was filed over privacy issues, the company stock still continued to rise.

Those are many reasons why we do not support Zoom. But there are also many reasons why we prefer BBB over Zoom. Besides, BBB has many great features that Zoom lacks:

1. Easily see who is speaking when their name appears above the presentation.
2. Chat messages will remain if you lose your connection or reload and rejoin the room.
3. Video is HD quality and you can easily focus on a persons webcam image.
4. Collaborative document writing on a shared Etherpad.
5. Easily share the presenter/admin role with others in the room.
6. Write closed captions in many languages, as well as change the language of the interface.
7. An interactive whiteboard for collaborative art with friends!

One huge advantage of free software, like BBB, is that you can usually find their issue queue where you can engage with the actual developers to report bugs and request feature enhancements. Here is a link to the BigBlueButton issue queue.

So, why do people keep using a platform like Zoom, even though there are many features in BigBlueButton that are much better?

There is very little publicity for free software and not many know it exists and that there are alternative solutions. You can find some great suggestions of software and switch to it by using this site called switching.software. The marketing budget for Zoom is large and leads you to believe it has everything you will need. Sadly their budget grows larger everyday with the money people pay for subscriptions to the platform. As a result, many people go with it as it is already used by their friends and colleagues, even though there are reports of irresponsible behavior by the Zoom Corporation. This is why the New York school system does not use Zoom and many organizations are following suit. The company gives people a false sense of security as it is widely used and very popular.

Of course, there are reasons to avoid other proprietary chat platforms too...

Agaric offers BigBlueButton for events and meetings. Check out our fun BBB website at CommunityBridge and test drive the video chat yourself!

If this discussion interests you, please share your thoughts with us in the comments.

Looking to learn more about problems with Zoom? There are a lot of articles about Zoom scandals.

If you need to leak a zoom video without being identified, you need to be very careful!
Zoom censorship of Palestine seminars sparks fight over academic freedom
Zoom encryption is not suited for secrets
Zoom is unclear about whether or not the app is actually end-to-end encrypted, which it isn't.
And, yes - the wikipedia page has a good summary of some issues
Stallman provides resources on what is bad about Zoom also

Looking to learn more about protecting your privacy online? These links have some helpful information and videos for tech-savvy people and organic folks alike!

2021 could be the year we all begin to STOP supporting the Corporations that oppress us.

Special thanks to Keegan Rankin for edits!

Drupal toolbar not working in dev environments in Firefox? Here's why.

2021 January 19

Drupal's toolbar second level of menu options and dropdown not showing? Look for "Uncaught DOMException: The quota has been exceeded." errors, as viewable in the Firefox web console. If you see them, the problem is likely due to sites sharing a top-level domain—which is likely if you are using a local development environment like DDEV, and you working on way too many sites at once—combined with a pretty bad Firefox bug that will be fixed in the next release.

To quote Nathan Monfils:

Everything from your public domain (abc.tld) counts against your quota, even if it is in a seemingly unrelated subdomain (e.g. my-app.example.com and intranet.example.com).

The quota is not recomputed properly, requiring a firefox restart after clearing your data on other subdomains

Note this may affect all sorts of applications, not just Drupal, when you have them running on multiple subdomains of the same top-level domain. So this isn't just about local development environments (and i dislike that DDEV shares their own top-level domain across all the instances you are working on, and while it can be changed i've accepted its way of doing things so i'm on the same page with other developers by default).

Sure, closing more tabs and restarting Firefox could (predictably) have fixed this—and a lot else that's wrong with me, according to everyone i know—but why do that when i can open more tabs and learn precisely how broken everything around me really is?

I am very happy the bug is fixed and this blog post will be obsolete in mere days! Usually this sort of technical noodlings get relegated to our raw notes, currently hosted through GitLab, but figured at least a few other Drupal developers would want to know what has been going on with their toolbars.

Image credit: "Too Many Tabs" by John Markos O'Neill is licensed with CC BY-SA 2.0.

Upgrade from Drupal 6 or 7 when it is right for you: In the era of modern Drupal, release cycles of major versions have only minor importance

2021 January 13

Upgrade from Drupal 6 or 7 when it is right for you

In the era of modern Drupal, release cycles of major versions have only minor importance

TLDR: Drupal 7 has a much longer lifespan than the (already pushed back) official date, and Drupal 8 has an essentially infinite lifespan because it updates in-place to Drupal 9 easily (and the same will be true of Drupal 10, 11, ∞.). There's no reason to rush an upgrade— but there's no reason to wait either.

That's the short version.

A client recently wrote to Agaric about Drupal 7 / Drupal 8 / Drupal 9 project planning questions:

With the EOL for Drupal 7 in Nov of 2022, and the EOL for Drupal 8 in Nov 2021, is there a reason we should move a D7 site to D8 at all this year? Seems like we might want to move directly to D9? We don’t want to feel pushed up against a wall with a “new” site build in Drupal 8, if we can limp along in D7 for a couple more years while we develop a D9 site with a longer lifespan. I’m wondering if you might have time to discuss pros and cons briefly so we can get a good plan together for moving forward.

I started typing and almost did not stop:

No one believes me when i say this, but i repeat my assurance that Drupal 7 will be well-supported commercially until 2030 or later (Drupal 6, released in 2008, still has semi-official long term support until at least February 24th, 2022— and Drupal 7 has a larger install base than Drupal 6 ever did, and currently has the largest install base of any version of Drupal by far, with more than half a million tracked installs.

Drupal 7 will be supported by the community for a long time. You do not have to feel pushed to a new version, like, ever.

We do recommend moving directly to Drupal 9 (which was released on June 3rd of 2020), however:
Moving to Drupal 8 or to Drupal 9 is much the same. Drupal 8 starts what i call the "modern Drupal" era. Whereas for going from Drupal 5 to 6 or 6 to 7 or 7 to 8 broke backward compatibility and might as well be a full rebuild (so we would often recommend hopping a version, say, stay on Drupal 6 and wait for Drupal 8 to be ready) going from Drupal 8 to 9 is closer to going from Drupal 8.8 to 8.9— an in-place upgrade from 8.9 to 9.0. Going from 9 to 10 will work the same, and that's the plan and promise for Drupal 8 on out.
All that said, if anything significant needs fixing on your current Drupal 7 site, or you are looking to make any improvements, you'll want to do that on Drupal 8+ or Drupal 8/9 as we phrased it back when Drupal 9 was still a pretty recent release, but now we can just say Drupal 9— or, as i call it to emphasize the decreased importance of major version numbers, modern Drupal.

Agaric is always happy to discuss more! Mostly what i'm saying here is the useful things to talk about are the specific goals for the sites—when you want to accomplish what—because the official support cycles are a distraction in the current context of Drupal. So make sure your current site is maintained, but take your time to get clear on your objectives, and contact Agaric or the Drupal professionals of your choice when you think it might make sense to upgrade your site into the era of modern Drupal.

Community-managed categories

2021 January 01

Community-managed categories is an idea from just about the beginning of my time as a web developer. As "Community-managed taxonomy" it was my submission to the 2007 Summer of Code, barely a couple years into my time as a Drupal developer.

The Drupal module Community Managed Taxonomy, or CMT, variously known as Community Managed Categories, seeks to bring the possibility of mass participation to categorization (taxonomy) and therefore potentially site structure.

As the project page put it:

Community-managed taxonomy (CMT) opens categorization of content to the site's community. Users can influence both what terms nodes are tagged with and how these terms are themselves organized.

It can also be used to make structured tags on the fly. Users do not need to be logged in to make or propose terms for content.

I very much hope to get back to this work. I never did get the module working—Agaric was a new company and my father became ill and was killed by the hospital that summer, and my mentor and i were not the best match—and after failing to complete the module and get the second stipend, the time and circumstance has not yet returned. Please contact us if you may be able to help make the time right in 2021!

This capability to allow a large community to coordinate in categorizing may be more salient when democratically-moderated mass communication is made possible. That's a goal i have pursued even longer, and i think it is more important and possibly logically prior to community-managed taxonomy. First we need community-managed communication, so that the resources we build together can reach, be distributed to, the people who should know. But building community-managed communication may bring up the need for community-managed categories directly, too— how do we decide what the groups are that we can manage communication within? That's a job for community-managed taxonomy, probably.

It is all tied up to what I have been aiming for for decades. This description is from 2008:

People have always needed something better than mailing lists— or other communication tools as they exist now. We need something that can reach millions of people (or billions– everyone) and still be open to everyone on an equal basis. Reaching everyone means filtering to reduce quantity and increase quality. Staying open to everyone means that the filtering must not be controlled by any group, must in some true sense belong to everyone.

The Internet has this potential. (The issue of access remains crucial, but is separate from helping the Internet reach closer to its potential for people who do have access.)

People Who Give a Damn is incorporated as a nonprofit organization to connect, without interference and without wasting anyone's time, everyone who gives a damn.

As the first technique to achieve this, anyone signed up to receive messages in a network can submit a message to be sent. The message will be publicly available immediately, but it will be moderated by a random sample of other people in the network pulled to serve jury duty. If they decide it is important enough to send to everyone in the network, it is sent to everyone. If not, the message will have more limited distribution to the sender's personal contacts and possibly to groups within the overall network to which the sender belongs.

This is simple. Yet it will make possible horizontal communication, not top-down few-to-many broadcasts, that is also mass communication. We need horizontal mass communication because we need mass cooperation and collaboration. It is possible with current technology, and necessary for the well-being of ourselves, our friends and family, our fellow humans, our Earth. Anyone interested in updates on progress or how they can help, please contact me.

Double-edged Raiser: Past time to ditch Blackbaud

2020 August 03

Double-edged Raiser

Past time to ditch Blackbaud

I have watched in sadness and sometimes anger as large non-profit after large non-profit collectively poured enough money into Raiser's Edge and other Blackbaud licenses and consulting services to fund many feature enhancements for the main FLOSS alternative, CiviCRM— improvements which would then be free for everyone, forever.

I have never met anyone who actually likes Blackbaud products and services. However, many organizations felt they were the only safe option, in the sense of claiming to have everything an enterprise needs.

Now, Blackbaud failed to secure its servers sufficiently and large amounts of its clients' donor data, including personally identifying information, was obtained in a ransomware attack. This was back in May. Blackbaud ultimately paid the ransomer to allegedly destroy the data they obtained— and only late in July finally told their customers what happened.

As the American Civil Liberties Union wrote to all its supporters, current and past (including myself), this is a rotten situation:

In all candor, we are frustrated with the lack of information we've received from Blackbaud about this incident thus far. The ACLU is doing everything in our power to ascertain the full nature of the breach, and we are actively investigating the nature of the data that was involved, details of the incident, and Blackbaud's remediation plans.

We are also exploring all options to ensure this does not happen again, including revisiting our relationship with Blackbaud.

Fortunately, none of Agaric's clients are affected. But we hope everyone using or considering using Blackbaud and other proprietary services for their most important data will look at free/libre open source solutions. Code you (or your technology partner) can see and contribute to means you truly can do anything. And if you put aside the money that would be gouged out of your organization by the eTapestry, Kintera, and Convio-swallowing monopolist Blackbaud, you probably can afford to.

At Agaric, we have recently been working with CiviCRM more recently (building on experience dating back fifteen years!) and we know our friends at Palante Technology Cooperative and myDropWizard are well-versed in CiviCRM, as are many others. Please consider this when weighing your options for maintaining a strong, ethical relationship with your supporters, and let us know if you have any thoughts or questions!

Are you building communities of practice for community health in this time of public recognition of injustice?

2020 July 16

January 25, 2021

Blog Update: Keep this conversation alive!

In this post, we call out for "Birds of a feather" to join us at DrupalCon, which has come and gone. However, this conversation remains relevant to our political condition and relevant to our work! Our scientific and government entities must continue to increasingly acknowledge racism as a public health threat. We believe that harnessing the power of data within our own communities is a path to the change that we want to see. Please help us keep this post and the discussion it provokes alive and circulating!

Original Post

The struggle to take seriously the impact of racism on public health during the covid-19 crisis spotlights the importance of health communities of practice and their need to have high-quality, data-driven discussions.

Environmental racism was forced into the national conversation by Flint, Michigan years ago and COVID-19 has made long-standing problems of racism in healthcare provision impossible to ignore. The rebellion against police repression, which started in Minneapolis, is itself a reaction to the public health issues caused by policing. This reaction was predictable, and Minneapolis resident D.A. Bullock, indeed, predicted it.

So, moving forward...

How do we have data-informed conversations effectively within our communities?

How do we expand them to include more health professionals and community members from outside of what is currently recognized as the healthcare industry?

We need resolute answers to both of these questions.

Thanks to a client giving us a shout out, Agaric has come to acknowledge our experience building for health communities and the role that we are able to take in this conversation. We would love to talk and learn with others more deeply about these issues. We're hosting a "Birds of a Feather" (people interested in the same topic coming together) at DrupalCon Global today at 3:15 Eastern Time to talk about facilitating discussion among healthcare practitioners, researchers, and the public.

Here are some more questions to get you thinking!

What are the next steps for healthcare workers and researchers? What are the next steps for any person who cares about our communities? How do we move important conversations into the public realm sustainably? The stakes for both well-informed and broad-based discussion are clearer than ever. We know pressure, policy, and practice are what make change; what is our role?

Please leave your comments even if you can't join us today!

Drupal migrations reference: List of configuration options for destination plugins

2020 July 15

In the previous article we provided a reference of available configuration options for migrate source plugins. In today’s article we are doing something similar for destination plugins. We will present a reference of available configuration options for migrate destination plugins provided by Drupal core and some contributed modules. Knowing which options are available might require some Drupal development knowledge. By providing this reference it should make the process of writing migrations easier.

For each migrate destination plugin we will present: the module that provides it, the class that defines it, the class that the plugin extends, and any inherited options from the class hierarchy. For each plugin configuration option we will list its name, type, a description, and a note if it is optional.

DestinationBase (abstract class)

Module: Migrate (Drupal Core)
Class: Drupal\migrate\Plugin\migrate\destination\DestinationBase
Extends: Drupal\Core\Plugin\PluginBase

This abstract class is extended by many migrate destination plugins. This means that the provided configuration keys apply to any destination plugin extending it.

List of configuration keys:

destination_module: An optional string value. Identifies the module handling the destination data. If not set, the Migrate API tries to read the value from the destination plugin definition.

Entity (abstract class)

Module: Migrate (Drupal Core) Plugin ID: entity:$entity_type See below.
Class: Drupal\migrate\Plugin\migrate\destination\Entity
Extends: Drupal\migrate\Plugin\migrate\destination\DestinationBase
Inherited configuration options: destination_module.
Related article: Drupal migrations reference: List of properties per content entity

This abstract class is extended by migrate destination plugins that want to import entities. It uses the MigrateEntity derivative to handle both content and configuration entities. The derivative sets the plugin ID to entity:$entity_type where $entity_type is the machine name of the entity. For example, entity:node and entity:node_type.

By default, content entities are handled by the EntityContentBase class while configuration entities use EntityConfigBase. Some entities like user (content) and node type (configuration) use specific classes for the import operation. The DefaultPluginManager::getDefinitions method triggers the search for classes that will override the default for a particular plugin ID. The override ultimately happens in the DerivativeDiscoveryDecorator::getDerivatives method.

In addition to the keys provided in the parent class chain, this abstract class provides the following configuration keys, which will be available to its derivatives:

default_bundle: An optional string value. Gets the bundle for the row taking into account the default. If not present, the bundle entity key should be set in the process section for entities that can have more than one bundle. For example, the type property for nodes, the vid property for taxonomy terms, and the bundle property for media entities.

EntityContentBase

Module: Migrate (Drupal Core) Plugin ID: entity:$entity_type See below.
Class: Drupal\migrate\Plugin\migrate\destination\EntityContentBase
Extends: Drupal\migrate\Plugin\migrate\destination\Entity
Inherited configuration options: destination_module and default_bundle.

This class is used to handle import operations for all content entities, unless a specific class exists for the plugin ID being used. For example, nodes are handled by this class, but users leverage the EntityUser class. The MigrateEntity derivative sets the plugin ID to entity:$entity_type where $entity_type is the machine name of the content entity. For example, entity:node, entity:user, entity:taxonomy_term, entity:file, entity:media, entity:comment, entity:block_content, and entity:contact_message.

In addition to the keys provided in the parent class chain, this class provides the following configuration keys:

translations: An optional boolean value. It indicates if the entity is translatable, defaults to FALSE. If set to TRUE, the corresponding langcode entity key will be added to the list of destination IDs that uniquely identifies each record. If the migration itself does not provide a value for the langcode property, the site’s default language is used.
overwrite_properties: An optional array of string values. It is a list of properties or fields in the destination entity that will be overwritten if an entity with the same ID already exists. Any properties that are not listed will not be overwritten. Refer to the class documentation for an example.
validate: An optional boolean value. It indicates whether an entity should be validated, defaults to FALSE. This was introduced in Drupal 8.8 and can be used to prevent invalid entities from being migrated. For example, a node with an empty title would fail validation. A required field that is not set by the migration will trigger a validation error, unless the field is configured to have a default value. Similarly, an integer field with minimum and maximum values will trigger an error if a value outside that range tries to be imported. Field API validations are triggered when this configuration option is set to TRUE.

EntityUser

Module: User (Drupal Core) Plugin ID: entity:user See below.
Class: Drupal\user\Plugin\migrate\destination\EntityUser
Extends: Drupal\migrate\Plugin\migrate\destination\EntityContentBase
Inherited configuration options: destination_module, default_bundle, translations, overwrite_properties, and validate.

This class provides a destination plugin for migrating user entities. It performs extra checks like preventing that the password for user 1 is overridden by the migration.

In addition to the keys provided in the parent class chain, this abstract class provides the following configuration keys:

md5_passwords: An optional boolean value. If set to TRUE, the pass (password) property of the user entity is assumed to contain an MD5 hash. The passwords will be salted and re-hashed before they are saved to the destination Drupal database.

EntityRevision

Module: Migrate (Drupal Core) Plugin ID: entity_revision:$entity_type See below.
Class: Drupal\migrate\Plugin\migrate\destination\EntityRevision
Extends: Drupal\migrate\Plugin\migrate\destination\EntityContentBase
Inherited configuration options: destination_module, default_bundle, translations, overwrite_properties, and validate.

This class provides an entity revision destination plugin. Only revisionable entities, those that define a revision entity key, can use this destination plugin. It uses the MigrateEntityRevision derivative which sets the plugin ID to entity_revision:$entity_type where $entity_type is the machine name of the entity. For example, entity_revision:node whose revision key is vid and entity_revision:block_content whose revision key is revision_id.

Entity revisions can only be migrated after the entity to which they belong has been migrated. For example, revisions of a given node (entity_revision:node destination migration) can be migrated only after the current version of that node (entity:node destination migration) has been imported.

EntityReferenceRevisions

Module: Entity Reference Revisions module Plugin ID: entity_reference_revisions:$entity_type See below.
Class: Drupal\entity_reference_revisions\Plugin\migrate\destination\EntityReferenceRevisions
Extends: Drupal\migrate\Plugin\migrate\destination\EntityRevision
Inherited configuration options: destination_module, default_bundle, translations, overwrite_properties, and validate.
Related article: Introduction to paragraphs migrations in Drupal

This class provides an entity revision revision destination plugin. It uses the MigrateEntityReferenceRevisions derivative which sets the plugin ID to entity_reference_revisions:$entity_type where $entity_type is the machine name of the entity. For example, entity_reference_revisions:node and entity_reference_revisions:paragraph. For example, entity_reference_revisions:node whose revision key is vid and entity_reference_revisions:paragraph whose revision key is revision_id.

This is the destination plugin used for migrating Paragraphs. Entity reference fields are no longer supported to reference Paragraphs. Instead, entity entity reference revisions must be used. Therefore, this class is used for paragraphs migrations with the entity_reference_revisions:paragraph plugin ID. See this article for an example on how to migrate paragraphs.

In addition to the keys provided in the parent class chain, this abstract class provides the following configuration keys:

new_revisions: An optional boolean value. Flag to indicate if a new revision should be created instead of updating a previous default record. Only applicable when providing an entity id without a revision_id. Defaults to FALSE.

EntityConfigBase

Module: Migrate (Drupal Core) Plugin ID: entity:$entity_id See below.
Class: Drupal\migrate\Plugin\migrate\destination\EntityConfigBase
Extends: Drupal\migrate\Plugin\migrate\destination\Entity
Inherited configuration options: destination_module and default_bundle.

This class is used to handle import operations for all configuration entities, unless a specific class exists for the plugin ID being used. For example, taxonomy vocabularies are handled by this class, but node types leverage the EntityNodeType class. The MigrateEntity derivative sets the plugin ID to entity:$entity_type where $entity_type is the machine name of the content entity. For example, entity:node_type, entity:user_role, entity:taxonomy_vocabulary, entity:block, entity:comment_type, entity:block_content_type, entity:contact_form, entity:date_format.

In addition to the keys provided in the parent class chain, this abstract class provides the following configuration keys:

translations: An optional boolean value. if TRUE, the destination will be associated with the langcode provided by the source plugin. Defaults to FALSE. For example: en for English, es for Spanish, and fr for French.

EntityNodeType

Module: Node (Drupal Core) Plugin ID: entity:node_type
Class: Drupal\node\Plugin\migrate\destination\EntityNodeType
Extends: Drupal\migrate\Plugin\migrate\destination\EntityConfigBase
Inherited configuration options: destination_module, default_bundle, and translations.

This class is used to import node types. It does not take extra configuration options. The plugin overrides the import method to attach the body field to the imported content type. This depends on the presence of certain destination properties in the imported row. That is, the following properties needs to be mapped in the process section of the migration:

create_body: An optional boolean value. If TRUE, a body field will be added to the content type.
create_body_label: An optional string value. If set and create_body is TRUE, the value for this destination property will be used as the label of the body field.

Config

Module: Migrate (Drupal Core) Plugin ID: config
Class: Drupal\migrate\Plugin\migrate\destination\Config
Extends: Drupal\migrate\Plugin\migrate\destination\DestinationBase
Inherited configuration options: destination_module.

This class persists data to the configuration management system. In addition to the keys provided in the parent class chain, this class provides the following configuration keys:

store null: An optional boolean value. If TRUE, when a property is NULL, the NULL value is stored. Otherwise, the default value is used. Defaults to FALSE. Note that there is a space character in the configuration name.
translations: An optional boolean value. if TRUE, the destination will be associated with the langcode provided by the source plugin. Defaults to FALSE.

Additionally, the plugin expects certain destination properties in the imported row. That is, the following properties needs to be mapped in the process section of the migration:

config_name: A string value. The machine name of the configuration. For example: node.settings, node.type.article, user.settings, system.site, and core.extension.
langcode: An optional string value. The language code of the configuration. For example: en for English, es for Spanish, and fr for French.

Table

Module: Migrate Plus Plugin ID: table
Class: Drupal\migrate_plus\Plugin\migrate\destination\Table
Extends: Drupal\migrate\Plugin\migrate\destination\DestinationBase
Inherited configuration options: destination_module.

This class allows you to write directly to a table not registered with Drupal Schema API. See this test for an example on how to use this plugin.

In addition to the keys provided in the parent class chain, this class provides the following configuration keys:

database_key: An string value. Key for the database connection used for inserting records. See this documentation page for more information on database connection keys. We also covered the topic when explaining the SqlBase source plugin.
table_name: An string value. Name of the table where records will be imported.
batch_size: An optional integer value. Maximum number of rows to insert in one query. Defaults to 1.
id_fields: An associative array value. Fields used to uniquely identify table rows. At least one field is required. See the class’s docblock for an example of the expected structure.
fields: An optional associative array value. Mapping of column names to values set in the process section.

Available configuration for other migrate destination plugins

In Drupal core itself there are around 50 migrate destination plugins. And many more are made available by contributed modules. It would be impractical to document them all here. To get a list by yourself, load the plugin.manager.migrate.destination service and call its getDefinitions() method. This will return all migrate destination plugins provided by the modules that are currently enabled on the site. This Drush command would get the list:

# List of migrate destination plugin definitions. $ drush php:eval "print_r(\Drupal::service('plugin.manager.migrate.destination')->getDefinitions());" # List of migrate destination plugin ids. $ drush php:eval "print_r(array_keys(\Drupal::service('plugin.manager.migrate.destination')->getDefinitions()));"

To find out which configuration options are available for any destination plugin consider the following:

Find the class that defines the plugin and find out which configuration values are read. Some plugins even include the list in the docblock of the class. Search for a pattern similar to $this->configuration['option_name'] or $configuration['option_name']. The plugins can be found in the Drupal\module_name\Plugin\migrate\destination namespace. The class itself would be in a file under the /src/Plugin/migrate/destination/ directory of the module.
Look up in the class hierarchy. The destination plugin can use any configuration set directly in its definition class and any parent class. There might be multiple layers of inheritance.
Check if the plugin requires the presence of specific destination properties to be set in the process section. This is usually documented in the class’ docblock, but you can also look for code like $row->getDestinationProperty('property_name').

What did you learn in today’s article? Did you know that migrate destination plugins can inherit configuration keys from their class hierarchy? Were you aware that there are so many destination plugins? Other than the ones listed here, which destination plugins have you used? Please share your answers in the comments. Also, we would be grateful if you shared this article with your friends and colleagues.

Drupal migrations reference: List of configuration options for source plugins

2020 July 13

In a previous article we explained the syntax used to write Drupal migration. We also provided references of subfields and content entities' properties including those provided by the Commerce module. This time we are going to list the configuration options of many migrate source plugins. For example, when importing from a JSON file you need to specify which data fetcher and parser to use. In the case of CSV migrations, the source plugin configuration changes depending on the presence of a headers row. Finding out which options are available might require some Drupal development knowledge. To make the process easier, in today’s article we are presenting a reference of available configuration options for migrate source plugins provided by Drupal core and some contributed modules.

For each migrate source plugin we will present: the module that provides it, the class that defines it, the class that the plugin extends, and any inherited options from the class hierarchy. For each plugin configuration option we will list its name, type, a description, and a note if it is optional.

SourcePluginBase (abstract class)

Module: Migrate (Drupal Core)
Class: Drupal\migrate\Plugin\migrate\source\SourcePluginBase
Extends: Drupal\Core\Plugin\PluginBase

This abstract class is extended by most migrate source plugins. This means that the provided configuration keys apply to any source plugin extending it.

List of configuration keys:

skip_count: An optional boolean value. If set, do not attempt to count the source. This includes status and import operations.
cache_counts: An optional boolean value. If set, cache the source count. This saves time calculating the number of available records in the source when a count operation is requested.
cache_key: An optional string value. Uniquely named cache key used for cache_counts.
track_changes: An optional boolean value. If set, the Migrate API will keep a hash of the source rows to determine whether the incoming data has changed. If the hash is different the record is re-imported.
high_water_property: An optional array value. If set, only content with a higher value will be imported. The value is usually a timestamp or serial ID indicating what was the last imported record. This key is configured as an associate array. The name key indicates the column in the source that will be used for the comparison. The alias key is optional and if set it serves as a table alias for the column name.
source_module: An optional string value. Identifies the system providing the data the source plugin will read. If not set, the Migrate API tries to read the value from the source plugin annotation. The source plugin itself determines how the value is used. For example, Migrate Drupal's source plugins expect source_module to be the name of a module that must be installed and enabled in the source database.

The high_water_property and track_changes are mutually exclusive. They are both designed to conditionally import new or updated records from the source. Hence, only one can be configured per migration definition file.

SqlBase (abstract class)

Module: Migrate (Drupal Core)
Class: Drupal\migrate\Plugin\migrate\source\SqlBase
Extends: Drupal\migrate\Plugin\migrate\source\SourcePluginBase
Inherited configuration options: skip_count, cache_counts, cache_key, track_changes, high_water_property, and source_module.

This abstract class is extended by migrate source plugins whose data may be fetched via a database connection. This means that the provided configuration keys apply to any source plugin extending it.

In addition to the keys provided in the parent class chain, this abstract class provides the following configuration keys:

key: An optional string value. The database key name. Defaults to 'migrate'.
target: An optional string value. The database target name. Defaults to 'default'.
database_state_key: An optional string value. Name of the state key which contains an array with database connection information. The Migrate API will consult the States API using the provided key. The returned value should be an associative array with at least two keys: key and target to determine which database connection to use. A third key database can also be included containing database connection information as seen in the snippet below.
batch_size: An optional integer value. Number of records to fetch from the database during each batch. If omitted, all records are fetched in a single query.
ignore_map: An optional boolean value. Source data is joined to the map table by default to improve performance. If set to TRUE, the map table will not be joined. Using expressions in the query may result in column aliases in the JOIN clause which would be invalid SQL. If you run into this, set ignore_map to TRUE.

To explain how these configuration keys are used, consider the following database connections:

<?php $databases['default']['default'] = [ 'database' => 'drupal-8-or-9-database-name', 'username' => 'drupal-8-or-9-database-username', 'password' => 'drupal-8-or-9-database-password', 'host' => 'drupal-8-or-9-database-server', 'port' => '3306', 'namespace' => 'Drupal\\Core\\Database\\Driver\\mysql', 'driver' => 'mysql', ]; $databases['migrate']['default'] = [ 'database' => 'drupal-6-or-7-database-name', 'username' => 'drupal-6-or-7-database-username', 'password' => 'drupal-6-or-7-database-password', 'host' => 'drupal-6-or-7-database-server', 'port' => '3306', 'namespace' => 'Drupal\\Core\\Database\\Driver\\mysql', 'driver' => 'mysql', ];

This snippet can be added to settings.php or settings.local.php. The $databases array is a nested array of at least three levels. The first level defines the database keys: default and migrate in our example. The second level defines the database targets: default in both cases. The third level is an array with connection details for each key/target combination. This documentation page contains more information about database configuration.

Based on the specified configuration values, this is how the Migrate API determines which database connection to use:

If the source plugin configuration contains database_state_key, its value is taken as the name of a States API key that specifies an array with the database configuration.
Otherwise, if the source plugin configuration contains key, the database configuration with that name is used.
Otherwise, load a fallback state key from the States API. The value that it tries to read is the global state key: migrate.fallback_state_key.
Otherwise, the database connection named migrate is used by default.
If all of the above steps fail, a RequirementsException is thrown.

Note that all values configuration keys are optional. If none is set, the plugin will default to use the connection specified under $databases['migrate']['default']. At least, set the key configuration even if the value is migrate. This would make it explicit which connection is being used.

DrupalSqlBase (abstract class)

Module: Migrate Drupal (Drupal Core)
Class: Drupal\migrate_drupal\Plugin\migrate\source\DrupalSqlBase
Extends: Drupal\migrate\Plugin\migrate\source\SqlBase
Inherited configuration options: skip_count, cache_counts, cache_key, track_changes, high_water_property, source_module, key, target, database_state_key, batch_size, and ignore_map.

This abstract class provides general purpose helper methods that are commonly needed when writing source plugins that use a Drupal database as a source. For example, check if the given module exists and read Drupal configuration variables. Check the linked class documentation for more available methods.

In addition to the keys provided in the parent class chain, this abstract class provides the following configuration key:

constants: An optional array value used to add module dependencies. The value is an associative array with two possible elements. The first uses the entity_type key to add a dependency on the module that provides the specified entity type. The second uses the module key to directly add a dependency on the specified module. In both cases, the DependencyTrait is used for setting the dependencies.

Warning: A plugin extending this abstract class might want to use this configuration key in the source definition to set module dependencies. If so, the expected keys might clash with other source constants used in the process pipeline. Arrays keys in PHP are case sensitive. Using uppercase in custom source constants might avoid this clash, but it is preferred to use a different name to avoid confusion.

This abstract class is extended by dozens of core classes that provide an upgrade path from Drupal 6 and 7. It is also used by the Commerce Migrate module to read product types, product display types, and shipping flat rates from a Commerce 1 database. The same module follows a similar approach to read data from an Ubercart database. The Paragraphs module also extends it to add and implement Configurable Plugin interface so it can import field collection types and paragraphs types from Drupal 7.

Config

Module: Migrate Drupal (Drupal Core). Plugin ID: d8_config
Class: Drupal\migrate_drupal\Plugin\migrate\source\d8\Config
Extends: Drupal\migrate_drupal\Plugin\migrate\source\DrupalSqlBase
Inherited configuration options: skip_count, cache_counts, cache_key, track_changes, high_water_property, source_module, key, target, database_state_key, batch_size, ignore_map, and constants.

This plugin allows reading configuration values from a Drupal 8 site by reading its config table.

In addition to the keys provided in the parent class chain, this plugin does not define extra configuration keys. And example configuration for this plugin would be:

source: plugin: d8_config key: migrate skip_count: true

In this case we are setting the key property from SqlBase to use the migrate default database connection. The skip_count from SourcePluginBase indicates that there is no need to count how many records exist in the source database before executing migration operations like importing them.

This plugin is presented to show that Drupal core already offers a way to migrate data from Drupal 8. Remember that there are dozens of other plugins extending DrupalSqlBase. It would be impractical to list them all here. See this API page for a list of all of them.

CSV

Module: Migrate Source CSV. Plugin ID: csv
Class: Drupal\migrate_source_csv\Plugin\migrate\source\CSV
Extends: Drupal\migrate\Plugin\migrate\source\SourcePluginBase
Inherited configuration options: skip_count, cache_counts, cache_key, track_changes, high_water_property, and source_module.

This plugin allows reading data from a CSV file. We used this plugin in the CSV migration example of the 31 days of migration series.

In addition to the keys provided in the parent class chain, this plugin provides the following configuration keys:

path: A string value. It contains the path to the CSV file. Starting with the 8.x-3.x branch, stream wrappers are supported. The original article contains more details on specifying the location of the CSV file.
ids: An array of string values. The column names listed are used to uniquely identify each record.
header_offset: An optional integer value. The index of record to be used as the CSV header and the thereby each record's field name. It defaults to zero (0) because the index is zero-based. For CSV files with no header row the value should be set to null.
fields: An optional associative array value. It contains a nested array of names and labels to use instead of a header row. If set, it will overwrite the column names obtained from header_offset.
delimiter: An optional string value. It contains a one character column delimiter. It defaults to a comma (,). For example, if your file uses tabs as delimiter, you set this configuration to \t.
enclosure: An optional string value. It contains one character used to enclose the column values. Defaults to double quotation marks (").
escape: An optional string value. It contains one character used for character escaping in the column values. It defaults to a backslash (\).

Important: The configuration options changed significantly between the 8.x-3.x and 8.x-2.x branches. Refer to this change record for a reference of how to configure the plugin for the 8.x-2.x.

For reference, below is the source plugin configuration used in the CSV migration example:

source: plugin: csv path: modules/custom/ud_migrations/ud_migrations_csv_source/sources/udm_photos.csv ids: [photo_id] header_offset: null fields: - name: photo_id label: 'Photo ID' - name: photo_url label: 'Photo URL'

Spreadsheet

Module: Migrate Spreadsheet. Plugin ID: spreadsheet
Class: Drupal\migrate_spreadsheet\Plugin\migrate\source\Spreadsheet
Extends: Drupal\migrate\Plugin\migrate\source\SourcePluginBase
Inherited configuration options: skip_count, cache_counts, cache_key, track_changes, high_water_property, and source_module.

This plugin allows reading data from Microsoft Excel and LibreOffice Calc files. It requires the PhpOffice/PhpSpreadsheet library and many PHP extensions including ext-zip. Check this page for a full list of dependencies. We used this plugin in the spreadsheet migration examples of the 31 days of migration series.

In addition to the keys provided in the parent class chain, this plugin provides the following configuration keys:

file: A string value. It stores the path to the document to process. You can use a relative path from the Drupal root, an absolute path, or stream wrappers.
worksheet: A string value. It contains the name of the one worksheet to process.
header_row: An optional integer value. This number indicates which row contains the headers. Contrary to CSV migrations, the row number is not zero-based. So, set this value to 1 if headers are on the first row, 2 if they are on the second, and so on.
origin: An optional string value. It defaults to A2. It indicates which non-header cell contains the first value you want to import. It assumes a grid layout and you only need to indicate the position of the top-left cell value.
columns: An optional array value. It is the list of columns you want to make available for the migration. In case of files with a header row, use those header values in this list. Otherwise, use the default title for columns: A, B, C, etc. If this setting is missing, the plugin will return all columns. This is not ideal, especially for very large files containing more columns than needed for the migration.
row_index_column: An optional string value. This is a special column that contains the row number for each record. This can be used as a unique identifier for the records in case your dataset does not provide a suitable value. Exposing this special column in the migration is up to you. If so, you can come up with any name as long as it does not conflict with header row names set in the columns configuration. Important: this is an autogenerated column, not any of the columns that comes with your dataset.
keys: An optional associative array value. If not set, it defaults to the value of row_index_column. It contains the "columns" that uniquely identify each record. The keys are column names as defined in the data_rows key. The values is an array with a single member with key 'type' and value a column type such as 'integer'. For files with a header row, you can use the values set in the columns configuration. Otherwise, use default column titles like A, B, C, etc. In both cases, you can use the row_index_column column if it was set.

Note that nowhere in the plugin configuration you specify the file type. The same setup applies for both Microsoft Excel and LibreOffice Calc files. The library will take care of detecting and validating the proper type.

For reference, below is the source plugin configuration used in the LibreOffice Calc migration example:

source: plugin: spreadsheet file: modules/custom/ud_migrations/ud_migrations_sheets_sources/sources/udm_book_paragraph.ods worksheet: 'UD Example Sheet' header_row: 1 origin: A2 columns: - book_id - book_title - 'Book author' row_index_column: 'Document Row Index' keys: book_id: type: string

SourcePluginExtension (abstract class)

Module: Migrate Plus
Class: Drupal\migrate_plus\Plugin\migrate\source\SourcePluginExtension
Extends: Drupal\migrate\Plugin\migrate\source\SourcePluginBase
Inherited configuration options: skip_count, cache_counts, cache_key, track_changes, high_water_property, and source_module.

This abstract class provides extra configuration keys. It is extended by the URL plugin (explained later) and by source plugins provided by other modules like Feeds Migrate.

In addition to the keys provided in the parent class chain, this abstract class provides the following configuration keys:

fields: An associative array value. Each element represents a field that will be made available to the migration. The following options can be set:
name: A string value. This is how the field is going to be referenced in the migration. The name itself can be arbitrary. If it contains spaces, you need to put double quotation marks (") around it when referring to it in the migration.
label: An optional string value. This is a description used when presenting details about the migration. For example, in the user interface provided by the Migrate Tools module. When defined, you do not use the label to refer to the field. Keep using the name.
selector: A string value. This is another XPath-like string to find the field to import. The value must be relative to the location specified by the item_selector configuration. In the example, the fields are direct children of the records to migrate. Therefore, only the property name is specified (e.g., unique_id). If you had nested objects or arrays, you would use a slash (/) character to go deeper in the hierarchy. This will be demonstrated in the image and paragraph migrations.
ids: An associative array value. It contains the "columns" that uniquely identify each record. The keys are column names as defined in the data_rows key. The values is an array with a single member with key 'type' and value a column type such as 'integer'.

See the code snippet for the Url plugin in the next section for an example of how these configuration options are used.

Url (used for JSON, XML, SOAP, and Google Sheets migrations)

Module: Migrate Plus. Plugin ID: url
Class: Drupal\migrate_plus\Plugin\migrate\source\Url
Extends: Drupal\migrate_plus\Plugin\migrate\source\SourcePluginExtension
Inherited configuration options: skip_count, cache_counts, cache_key, track_changes, high_water_property, source_module, fields, and ids.

This plugin allows reading data from URLs. Using data parser plugins it is possible to fetch data from JSON, XML, SOAP, and Google Sheets. Note that this source plugin uses other plugins provided by Migrate Plus that might require extra configuration keys in addition to the ones explicitly defined in the plugin class. Those will also be listed.

In addition to the keys provided in the parent class chain, this plugin provides the following configuration keys:

urls: An array of string values. It contains the source URLs to retrieve. If the file data fetcher plugin is used, the location can be a relative path from the Drupal root, an absolute path, a fully-qualified URL, or a stream wrapper. See this article for details on location of the JSON file. If the HTTP data fetcher plugin is used, the value can use any protocol supported by curl. See this article for more details on importing remote JSON files.
data_parser_plugin: A string value. It indicates which data parser plugin to use. Possible values provided by the Migrate Plus module are json, xml, simple_xml, and soap. If the Migrate Google Sheets module is installed, it is possible to set the value to google_sheets. Review the relevant articles in the 31 days of migration series to know more about how each of them are used.

The data parser plugins provide the following configuration keys:

item_selector: A string value. It indicates where in the source file lies the array of records to be migrated. Its value is an XPath-like string used to traverse the file hierarchy. Note that a slash (/) is used to separate each level in the hierarchy.
data_fetcher_plugin: A string value. It indicates which data parser plugin to use. Possible values provided by the Migrate Plus module are file and http.

The HTTP data fetcher plugins provide the following configuration keys:

headers: An associative array value. The key/value pairs represent HTTP headers to be sent when the request is made.
authentication: An associative array value. One of the elements in the array should be the plugin key which indicates the authentication plugin to use. Possible values provided by the Migrate Plus module are basic, digest, and oauth2. Other elements in the array will depend on the selected authentication plugin.

The basic and digest authentication plugins provide the following configuration keys:

username: A string value.
password: A string value.

The OAuth2 authentication plugin requires the sainsburys/guzzle-oauth2-plugin composer package to work. It provides the following configuration keys:

base_uri: A string value.
grant_type: A string value. Possible values are authorization_code, client_credentials, urn:ietf:params:oauth:grant-type:jwt-bearer, password, and refresh_token. Each of these might require extra configuration values.

The client credentials grant type requires the following configuration keys:

token_url: A string value.
client_id: A string value.
client_secret: A string value.

For configuration keys required by other grant types, refer to the classes that implement them. Read this article on adding HTTP request headers and authentication parameters for example configurations.

There are many combinations possible to configure this plugin. In the 31 days of migration series there are many example configurations. For reference, below is the source plugin configuration used in the local JSON node migration example:

source: plugin: url data_fetcher_plugin: file data_parser_plugin: json urls: - modules/custom/ud_migrations/ud_migrations_json_source/sources/udm_data.json item_selector: /data/udm_people fields: - name: src_unique_id label: 'Unique ID' selector: unique_id - name: src_name label: 'Name' selector: name - name: src_photo_file label: 'Photo ID' selector: photo_file - name: src_book_ref label: 'Book paragraph ID' selector: book_ref ids: src_unique_id: type: integer

EmbeddedDataSource

Module: Migrate (Drupal Core). Plugin ID: embedded_data
Class: Drupal\migrate\Plugin\migrate\source\EmbeddedDataSource
Extends: Drupal\migrate\Plugin\migrate\source\SourcePluginBase
Inherited configuration options: skip_count, cache_counts, cache_key, track_changes, high_water_property, and source_module.

This plugin allows the definition of data to be imported right inside the migration definition file. We used this plugin in many of the examples of the 31 days of migration series. It is also used in many core tests for the Migrate API itself.

In addition to the keys provided in the parent class chain, this abstract class provides the following configuration keys:

data_rows: An array of all the records to be migrated. Each record might contain an arbitrary number of key-value pairs representing "columns" of data to be imported.
ids: An associative array. It contains the "columns" that uniquely identify each record. The keys are column names as defined in the data_rows key. The values is an array with a single member with key 'type' and value a column type such as 'integer'.

Many examples of 31 days of migration series use this plugin. You can get the example modules from this repository. For reference, below is the source plugin configuration used in the first migration example:

source: plugin: embedded_data data_rows: - unique_id: 1 creative_title: 'The versatility of Drupal fields' engaging_content: 'Fields are Drupal''s atomic data storage mechanism...' - unique_id: 2 creative_title: 'What is a view in Drupal? How do they work?' engaging_content: 'In Drupal, a view is a listing of information. It can a list of nodes, users, comments, taxonomy terms, files, etc...' ids: unique_id: type: integer

This plugin can also be used to create default content when the data is known in advance. We often present Drupal site building workshops. To save time, we use this plugin to create nodes which are later used when explaining how to create Views. Check this repository for an example of this. Note that it uses a different directory structure to store the migrations as explained in this blog post.

ContentEntity

Module: Migrate Drupal (Drupal Core). Plugin ID: content_entity
Class: Drupal\migrate_drupal\Plugin\migrate\source\ContentEntity
Extends: Drupal\migrate\Plugin\migrate\source\SourcePluginBase
Inherited configuration options: skip_count, cache_counts, cache_key, track_changes, high_water_property, and source_module.

This plugin returns content entities from a Drupal 8 or 9 installation. It uses the Entity API to get the data to migrate. If the source entity type has custom field storage fields or computed fields, this class will need to be extended and the new class will need to load/calculate the values for those fields.

In addition to the keys provided in the parent class chain, this plugin provides the following configuration key:

entity_type: A string value. The entity type ID of the entities being migrated. This is calculated dynamically by the deriver so it is only needed if the deriver is not utilized, i.e., a custom source plugin.
bundle: An optionals string value. If set and the entity type is bundleable, only return entities of this bundle.
include_translations: An optional boolean value. If set, entity translations are included in the returned data. It defaults to TRUE.

For reference, this is how this plugin is configured to get all nodes of type article in their default language only:

source: plugin: content_entity:node bundle: article include_translations: false

Note: this plugin was brought into core in this issue copied from the Drupal 8 migration (source) module. The latter can be used if the source database does not use the default connection.

Table

Module: Migrate Plus. Plugin ID: table
Class: Drupal\migrate_plus\Plugin\migrate\source\Table
Extends: Drupal\migrate\Plugin\migrate\source\SqlBase
Inherited configuration options: skip_count, cache_counts, cache_key, track_changes, high_water_property, source_module, key, target, database_state_key, batch_size, and ignore_map.

This plugin allows reading data from a single database table. It uses one of the database connections for the site as defined by the options. See this test for an example on how to use this plugin.

In addition to the keys provided in the parent class chain, this plugin provides the following configuration key:

table_name: An string value. The table to read values from.
id_fields: An associative array value. IDMap compatible array of fields that uniquely identify each record.
fields: An array value. The elements are fields ("columns") present on the source table.

EmptySource (migrate module)

Module: Migrate (Drupal Core). Plugin ID: empty
Class: Drupal\migrate\Plugin\migrate\source\EmptySource
Extends: Drupal\migrate\Plugin\migrate\source\SourcePluginBase
Inherited configuration options: skip_count, cache_counts, cache_key, track_changes, high_water_property, and source_module.

This plugin returns an empty row by default. It can be used as a placeholder to defer setting the source plugin to a deriver. An example of this can be seen in the migrations for Drupal 6 and Drupal 7 entity reference translations. In both cases, the source plugin will be determined by the EntityReferenceTranslationDeriver.

In addition to the keys provided in the parent class chain, this plugin does not define extra configuration keys. If the plugin is used with source constants, a single row containing the constant values will be returned. For example:

source: plugin: empty constants: entity_type: node field_name: body

The plugin will return a single row containing 'entity_type' and 'field_name' elements, with values of 'node' and 'body', respectively. This is not very useful. For the most part, the plugin is used to defer the definition to a deriver as mentioned before.

EmptySource (migrate_drupal module)

Module: Migrate Drupal (Drupal Core). Plugin ID: md_empty
Class: Drupal\migrate_drupal\Plugin\migrate\source\EmptySource
Extends: Drupal\migrate\Plugin\migrate\source\EmptySource
Inherited configuration options: skip_count, cache_counts, cache_key, track_changes, high_water_property, and source_module.

By default, this plugin returns an empty row with Drupal specific config dependencies. If the plugin is used with source constants, a single row containing the constant values will be returned. These can be seen in the user_picture_field.yml and d6_upload_field.yml migrations.

In addition to the keys provided in the parent class chain, this abstract class provides the following configuration keys:

constants: An optional array value used to add module dependencies. The value is an associative array with one possible element. An entity_type key is used to add a dependency on the module that provides the specified entity type.

Available configuration for other migrate source plugins

In Drupal core itself there are more than 100 migrate source plugins, most of which come from the Migrate Drupal module. And many more are made available by contributed modules. It would be impractical to document them all here. To get a list by yourself, load the plugin.manager.migrate.source service and call its getFieldStorageDefinitions() method. This will return all migrate source plugins provided by the modules that are currently enabled on the site. This Drush command would get the list:

# List of migrate source plugin definitions. $ drush php:eval "print_r(\Drupal::service('plugin.manager.migrate.source')->getDefinitions());" # List of migrate source plugin ids. $ drush php:eval "print_r(array_keys(\Drupal::service('plugin.manager.migrate.source')->getDefinitions()));"

To find out which configuration options are available for any source plugin consider the following:

Find the class that defines the plugin and find out which configuration values are read. Some plugins even include the list in the docblock of the class. Search for a pattern similar to $this->configuration['option_name'] or $configuration['option_name']. The plugins can be found in the Drupal\module_name\Plugin\migrate\source namespace. The class itself would be in a file under the /src/Plugin/migrate/source/ directory of the module.
Look up in the class hierarchy. The source plugin can use any configuration set directly in its definition class and any parent class. There might be multiple layers of inheritance.
Check if the plugin offers some extension mechanism that would allow it to use other types of plugins. For example, the data fetcher, data parses, and authentication plugins provided by Migrate Plus. The Url migrate source plugin does this.

What did you learn in today’s article? Did you know that migrate source plugins can inherit configuration keys from their class hierarchy? Were you aware that there are so many source plugins? Other than the ones listed here, which source plugins have you used? Please share your answers in the comments. Also, we would be grateful if you shared this article with your friends and colleagues.

Drupal migrations reference: List of properties per content entity

2020 July 08

In a previous article we explained the syntax used to write Drupal migrations. When migrating into content entities, these define several properties that can be included in the process section to populate their values. For example, when importing nodes you can specify the title, publication status, creation date, etc. In the case of users, you can set the username, password, timezone, etc. Finding out which properties are available for an entity might require some Drupal development knowledge. To make the process easier, in today’s article we are presenting a reference of properties available in content entities provided by Drupal core and some contributed modules.

For each entity we will present: the module that provides it, the class that defines it, and the available properties. For each property we will list its name, field type, a description, and a note if the field allows unlimited values (i.e. it has an unlimited cardinality). The list of properties available for a content entity depend on many factors. For example, if the entity is revisionable (e.g. revision_default), translatable (e.g. langcode), or both (e.g. revision_translation_affected). The modules that are enabled on the site can also affect the available properties. For instance, if the “Workspaces” module is installed, it will add a workspace property to many content entities. This reference assumes that Drupal was installed using the standard installation profile and all modules that provide content entities are enabled.

It is worth noting that entity properties are divided in two categories: base field definitions and field storage configurations. Base field configurations will always be available for the entity. On the other hand, the presence of field storage configurations will depend on various factors. For one, they can only be added to fieldable entities. Attaching the fields to the entity can be done manually by the user, by a module, or by an installation profile. Again, this reference assumes that Drupal was installed using the standard installation profile. Among other things, it adds a user_picture image field to the user entity and body, comment, field_image, and field_tags fields to the node entity. For entities that can have multiple bundles, not all properties provided by the field storage configurations will be available in all bundles. For example, with the standard installation profile all content types will have a body field associated with it, but only the article content type has the field_image, and field_tags fields. If subfields are available for the field type, you can migrate into them.