The custom style section should be viewed as a CSS stylesheet for collapse-o-matic plugin.
Some examples of how I’ve used it include assigning ID’s to certain expand collapse shortcodes and then modifying the CSS.
Example:
[expand title="My Title" id="example"] This is my content [/expand]
Then in the custom style section:
#example { background-color: red; height: 50px; }
Hope this helps!