Child menu query

Hi,
Lets say that I have the menu

colors
>red
>green
>blue
animals
>dog
>cat
>horse

…in my primary menu in WP in that visual order.
When I navigate to cat I want just just the following in my left sidebar nav:
>dog
>cat
>horse
(in that visual order).
I know I can assign pages to parents when creating pages in WP and getting that to work is fine.
I want however to derive the child links from the primary menu view itself.
There is a widget that will give me the entire menu but I just want the child links from the menu itself.
Can that be done without having to to assign a page to a parent each time?
I want the visual menu appearance structure to determine what children the parent has in the sidebar.
Thanks